Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Natural Language Generation #13

Merged
merged 19 commits into from
Mar 16, 2020
Merged

Natural Language Generation #13

merged 19 commits into from
Mar 16, 2020

Conversation

guillaume-be
Copy link
Owner

Language generation pipeline implemented, based on the Transformer's generation script.

  • GPT ad GPT2 supported
  • Greedy decoding and beam search
  • implementation of top-k and nucleus (top-p) sampling
  • temperature setting
  • repetition penalty
  • repetitive tokens ban (based on Hugging Face and Fairseq's implementation)

@guillaume-be
initial structure for generation
1923af0
@guillaume-be
GPT2 download
c98b689
@guillaume-be
Added tests for GPT2
2d63fc3
@guillaume-be
Updated copyright
16a1059
@guillaume-be
Updated generation preparation
8f044b5
@guillaume-be
Model forward pass for generation & GPT2 fix
c7727b0
@guillaume-be
Added repetition penalty
dd3415b
@guillaume-be
Updated generation based on latest Transformers update
de86e57
@guillaume-be
Updated generation based on latest Transformers update (preparation)
61721ee
@guillaume-be
Initial implementation of banned tokens detection
b6e186b
@guillaume-be
banned tokens and min length implemented
2241f8c
@guillaume-be
Addition of top_k top_p sampling
c2ff52c
@guillaume-be
Update of input ids, stop if eos_id and update attention mask
6e61a54
@guillaume-be
End to end generation without beam search
e572cdb
@guillaume-be
Added tests for GPT2 and OpenAI GPT generation
2621e0d
@guillaume-be
beam search preparation
2dbdb2c
@guillaume-be
Beam population loop
5539e93
@guillaume-be
End-to-end generation completed, updated README, bumped version
64a01e3
@guillaume-be
End-to-end generation completed, updated README, bumped version
1efe107
@guillaume-be guillaume-be merged commit 7c6e711 into master Mar 16, 2020
@guillaume-be guillaume-be deleted the generation branch May 1, 2020 16:16
@sachaarbonel sachaarbonel mentioned this pull request Jun 28, 2020
Closed
@ycat3 ycat3 mentioned this pull request Feb 21, 2022
Closed
@SorooshMortazavi SorooshMortazavi mentioned this pull request Apr 9, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@guillaume-be