Skip to content

convert_to_hf_gptneox adds support to other neox models#61

Merged
csris merged 3 commits intotogethercomputer:mainfrom
LorrinWWW:conversion
Mar 29, 2023
Merged

convert_to_hf_gptneox adds support to other neox models#61
csris merged 3 commits intotogethercomputer:mainfrom
LorrinWWW:conversion

Conversation

@LorrinWWW
Copy link
Copy Markdown
Contributor

Previous conversion code assumes the model is neox-20b, and some config is hard-coded, i.e. 44 layers.
(also added a fp16 option.)

example:

python tools/convert_to_hf_gptneox.py \
    --config-name EleutherAI/pythia-6.9b-deduped \
    --ckpt-path ${PATH_CKPTS}/checkpoint_100 --save-path pythia-6.9b-finetuned \
    --n-stages 4 --n-layer-per-stage 8 --fp16

@LorrinWWW LorrinWWW changed the title support other neox models convert_to_hf_gptneox adds support to other neox models Mar 21, 2023
@ETERNALBLUEbullrun
Copy link
Copy Markdown

Would use (plus help with issues,) but for the lack of support for regular GPT models.

@csris
Copy link
Copy Markdown
Contributor

csris commented Mar 22, 2023

We need to update the README as well. @LorrinWWW, can you do that?

@LorrinWWW
Copy link
Copy Markdown
Contributor Author

@csris Sure I will do that

@LorrinWWW
Copy link
Copy Markdown
Contributor Author

Done

@814007450
Copy link
Copy Markdown

814007450 commented Mar 23, 2023 via email

@csris csris merged commit 5834060 into togethercomputer:main Mar 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants