SFT - Dataset Packing causes C10d timeout

[SuperSecureHuman]

Current DDP timeout by default is set to 30 min via HF trainer. In case of larger datasets, this causes timeout during the process of packing.

There are 2 options to over come this..

Either start the DDP/C10d after the packing is done

Or

Provide packing dataset as a function along with datasets - so that a packed dataset can be stored, and then loaded directly.

Please let me know if I am missing something, or doing something wrong.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.