how to sft support larger context length?

[lucasjinreal]

if set max_sequence_len to 4k, does the model able to do extrapolation automatically?

[jiangchengSilent]

change max_seq_len in script args, as well as max_position_embeddings in config.json in model file to larger value. But this needs more GPU memory. Consider to adjust related hyper-params such as batch_size

[lucasjinreal]

I mean, doesit has dynamic ntk or rope scaling inside code.....

[cArlIcon]

I mean, doesit has dynamic ntk or rope scaling inside code.....

No. Just modify max_position_embeddings in config.json. It can extend context length to 32K for Yi-34 and Yi-6.

[lucasjinreal]

Doesn't it will consume too much mem?

[jiangchengSilent]

Doesn't it will consume too much mem?

Yes it is. It takes a lot GPU mem or CPU mem if zero-offload is enabled. So a single node with 8 cards and 900GB CPU mem is not enough for such setting. Please try multi-node training