Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[EXAMPLE] Port Codegen model into alpa #774

Merged
merged 54 commits into from
Feb 7, 2023
Merged

[EXAMPLE] Port Codegen model into alpa #774

merged 54 commits into from
Feb 7, 2023

Conversation

BabyChouSr
Copy link
Contributor

@BabyChouSr BabyChouSr commented Nov 13, 2022 •
edited

Context: Trying to migrate codegen model into alpa

The main code in codegen_model.py comes from modeling_flax_gptj.py

@BabyChouSr
codegen model init
30a0d8c
@BabyChouSr
add codegen prefix to model
fe8cbb3
@BabyChouSr
add codegen into wrapper
2c5b964
@BabyChouSr
remove autoregressive
058477f
@BabyChouSr
add codegen to textgen
f63d2e8
@BabyChouSr
redo codegen configs inside model
8ad39c4
@BabyChouSr
rename params in codegen
d9a6fdf
@BabyChouSr
remove extra n
310f41f
@BabyChouSr
replace max target positions by max seq len
96e7ea1
@BabyChouSr
refactor loading params
cc5eb96
@BabyChouSr
replace decoder embed dims with hidden size
5ac2d58
@BabyChouSr
replace decoder layers with num_hidden_layers
b586029
@BabyChouSr
replace decoder attention heads with n_head
e79216b
@BabyChouSr
added some config params
9d353c1
@BabyChouSr
add deterministic for dropout
c0b66ea
@BabyChouSr
fix params loading
83aca90
@BabyChouSr
fix more config lines
2d1801c
@BabyChouSr
Merge remote-tracking branch 'origin/pr-codegen-llm' into pr-codegen-llm
d494212
@BabyChouSr
add out proj and dont use bias
aefd87e
@BabyChouSr
fix embeddings
ded6f15
@BabyChouSr
replace position ids with token_type
d95007a
@BabyChouSr
delete redundancies
6389f98
@BabyChouSr
fix code blocks to mirror gptj
eb9d589
@BabyChouSr
fix codegenattention
6a360ea
@BabyChouSr
fix param pass through
5a98c60
@BabyChouSr
fix mismatched shapes and transformer module
76858ea
@BabyChouSr
add position ids to params and fix precision
973ee8b
@BabyChouSr
Merge branch 'pr-codegen-llm' of https://github.com/BabyChouSr/alpa i…
d4d4cbb 
…nto pr-codegen-llm
@BabyChouSr
fix vocab size
de998c9
@BabyChouSr
remove layer and fix weights
c04291b
@merrymercy merrymercy assigned merrymercy and unassigned merrymercy Dec 23, 2022
@BabyChouSr BabyChouSr marked this pull request as ready for review February 7, 2023 16:55
@merrymercy
Copy link
Member

merrymercy commented Feb 7, 2023 •
edited

LGTM. I applied some small tweaks. Thanks for the contribution!

@merrymercy merrymercy merged commit bec4fae into alpa-projects:main Feb 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@merrymercy merrymercy merrymercy left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@BabyChouSr @merrymercy