how to use 200k model to do SFT

[guanghuixu]

only change --max_seq_len 204800 in the finetune/scripts/run_sft_Yi_6b.sh?

[cArlIcon]

200k=200000 in Yi model

[guanghuixu]

so just modify this position？

[jiangchengSilent]

change max_seq_len in script args, as well as max_position_embeddings in config.json in model file to 200k. But this will obviously need more GPU memory. A singe node with 8 cards may not be enough.
Consider using multi-node sft training by setting deepspeed -H /path-to-hostfile/hostfile main.py ........

[dachengai]

A singe node with 8 x A100 80G cards could sft 4k context ?

[jiangchengSilent]

A singe node with 8 x A100 80G cards could sft 4k context ?

By our experiments, it is okay to sft 4k with zero2+offload on a single node, and sft 16k with zero3+offload by properly setting deepspeed configs in utils/ds_utils.py

[dachengai]

A singe node with 8 x A100 80G cards could sft 4k context ?

By our experiments, it is okay to sft 4k with zero2+offload on a single node, and sft 16k with zero3+offload by properly setting deepspeed configs in utils/ds_utils.py

thanks

[dachengai]

A singe node with 8 x A100 80G cards could sft 4k context ?

By our experiments, it is okay to sft 4k with zero2+offload on a single node, and sft 16k with zero3+offload by properly setting deepspeed configs in utils/ds_utils.py

6B or 34B ?

[jiangchengSilent]

A singe node with 8 x A100 80G cards could sft 4k context ?

By our experiments, it is okay to sft 4k with zero2+offload on a single node, and sft 16k with zero3+offload by properly setting deepspeed configs in utils/ds_utils.py

6B or 34B ?

34b