Support for LLaMA

[zphang]

No description provided.

[StellaAthena]

@zphang where are we on this?

[zphang]

Other than the above note on LLaMAMLP, I can incorporate the necessary changes and update the PR.

[zphang]

Addressed comments. Please take another look. For further additions (e.g. guide to tuning a LLaMA model), I can do a separate PR.

[StellaAthena]

I'm confused by the "make_vocab_size_divisible_by" argument in the config file. Can you explain what it does? We generally assume that the data has been preprocessed including pre-tokenzied, and that's the point in time where it would make sense to have such a thing (and we do).

Otherwise, this looks good to me. @Quentin-Anthony, any thoughts?

[zphang]

make_vocab_size_divisible_by is an argument pre-existing in NeoX. If I understand correctly, NeoX determines the vocab size based on the tokenizer, and then pads it to a multiple of make_vocab_size_divisible_by for computational efficiency reasons. Setting it to 1 effectively disables the padding. The feature isn't LLaMA-specific, but I think this makes sense for a default config for LLaMA models.

[Quentin-Anthony]

I'm confused by the "make_vocab_size_divisible_by" argument in the config file. Can you explain what it does? We generally assume that the data has been preprocessed including pre-tokenzied, and that's the point in time where it would make sense to have such a thing (and we do).

Otherwise, this looks good to me. @Quentin-Anthony, any thoughts?

Looking good to me. I'm gonna run some tests then merge.

[CLAassistant]

All committers have signed the CLA.

[StellaAthena]

@zphang We need you to sign the CLA before merging this PR :)

[zphang]

Signed!

all comments appear resolved now

[Quentin-Anthony]

Works for all of my tests. Merging.

[DaoD]

@zphang Thanks for your work! It seems that there is no params.json in the llama checkpoint. Where can I get it? Thanks!

[wiio12]

@zphang Hi, thank you for your work! can you provide a readme explaining how to run llama in neox?

[wiio12]

@zphang Thanks for your work! It seems that there is no params.json in the llama checkpoint. Where can I get it? Thanks!

Hi @DaoD, this params.json file can be found in the offical checkpoint (It requires to fill the google form).

[DaoD]

@zphang Thanks for your work! It seems that there is no params.json in the llama checkpoint. Where can I get it? Thanks!

Hi @DaoD, this params.json file can be found in the offical checkpoint (It requires to fill the google form).

Thanks so much!

[DaoD]

@zphang Hi, thank you for your work! can you provide a readme explaining how to run llama in neox?

Yes. I have converted the checkpoint and tried to use the 6-7B's config to load it, but there are some missing keys and unexpected keys in the dict. Could you please provide a readme for using it?

[wiio12]

Hi @DaoD, I think llama should be loaded with the llama/7B.yml config file.

I post an issue that discusses problems that I came across when loading llama in Neox.

[DaoD]

@zphang I think we need a new convert_sequential_to_hf.py to convert the obatined model into HF style. Have you done something about this? Thanks!

[borghives]

@zphang in the tools/convert_raw_llama_weights_to_neox.py, how does one convert llama tokenizer?

• I have tried using the documented flag in the tool "--model_size tokenizer_only" (got error "assert model_size in NUM_SHARDS" when trying to run the convert_raw_llama_weights_to_neox.py)
• I have tried using the huggingface llama tokenizer.json file (got error missing "merge-file" when trying to run generate)

[DaoD]

@zphang in the tools/convert_raw_llama_weights_to_neox.py, how does one convert llama tokenizer?

• I have tried using the documented flag in the tool "--model_size tokenizer_only" (got error "assert model_size in NUM_SHARDS" when trying to run the convert_raw_llama_weights_to_neox.py)
• I have tried using the huggingface llama tokenizer.json file (got error missing "merge-file" when trying to run generate)

I think you do not need to convert llama tokenizer. Just set

"tokenizer_type": "SPMTokenizer",
"vocab-file": "/llama-7b-hf/tokenizer.model",

The tokenizer.model can be obtained from this link.

[DaoD]

I find another problem. The eod token's id of SentencePieceTokenizer (eos_token_id=0) is differnet from the orginal LlamaTokenizer (eos_token_id =1), which may cause some problems in training and inference.

[wiio12]

I find another problem. The eod token's id of SentencePieceTokenizer (eos_token_id=0) is differnet from the orginal LlamaTokenizer (eos_token_id =1), which may cause some problems in training and inference.

Hi @DaoD, have you tried to do inference with this model, can you generate reasonable text with it?

[DaoD]

I find another problem. The eod token's id of SentencePieceTokenizer (eos_token_id=0) is differnet from the orginal LlamaTokenizer (eos_token_id =1), which may cause some problems in training and inference.

Hi @DaoD, have you tried to do inference with this model, can you generate reasonable text with it?

Yes, but I do not use the inference code provided by gpt-neox. I just convert the model into Huggingface style, and use the HF function for generation. It seems correct.

[wiio12]

I find another problem. The eod token's id of SentencePieceTokenizer (eos_token_id=0) is differnet from the orginal LlamaTokenizer (eos_token_id =1), which may cause some problems in training and inference.

Hi @DaoD, have you tried to do inference with this model, can you generate reasonable text with it?

Yes, but I do not use the inference code provided by gpt-neox. I just convert the model into Huggingface style, and use the HF function for generation. It seems correct.

Thx! This helps a lot:)

[sxthunder]

Hello, can you convert a gpt-neox llama model to HF style?