Skip to content

Conversation

chenmoneygithub
Copy link
Contributor

@chenmoneygithub chenmoneygithub commented Mar 7, 2023

With the finetuning, the generated text stops properly at end token, and the writing is better.

To play with the preset, install this branch:

!pip install -q -U git+https://github.com/chenmoneygithub/keras-nlp.git@add-gpt2-news

then use the code below:

import keras_nlp

gpt2_lm = keras_nlp.models.GPT2CausalLM.from_preset("gpt2_base_en_cnn_dailymail")
gpt2_lm.generate(
    ["that's weird", "that's even weirder"],
    sampler="top_k",
    max_length=100,
)

Copy link
Member

@mattdangerw mattdangerw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice this seems useful to have. Just a few comments.

"merges_url": "https://storage.googleapis.com/keras-nlp/models/gpt2_extra_large_en/v1/merges.txt",
"merges_hash": "75a37753dd7a28a2c5df80c28bf06e4e",
},
"gpt2_base_en_news": {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the most similar checkpoint we have here is the sst2 dataset for bert. So maybe we should stick to the more precise naming and call this, gpt2_base_en_cnn_dailymail?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure!

Copy link
Member

@mattdangerw mattdangerw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving! Let's update our file paths before we merge.

"max_sequence_length": 1024,
},
"preprocessor_config": {},
"weights_url": "https://storage.googleapis.com/keras-nlp/models/gpt2_base_en_news/model.h5",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update URLs to the new path before we merge this. Also remember to add a /v1/ to the path as above, so we can update our checkpoints in the future.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done!

@chenmoneygithub chenmoneygithub merged commit a513f03 into keras-team:master Mar 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants