We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to pre-build the dataset's index ?
I want to avoid using compute node for this task:
> WARNING: could not find index map files, building the indices on rank 0 ... > elasped time to build and save doc-idx mapping (seconds): 270.614145
The text was updated successfully, but these errors were encountered:
you can use --data-cache-path to specify where you want to cache. And precompute it using a single node.
Megatron-LM/megatron/training/arguments.py
Lines 1349 to 1350 in 9de386d
Sorry, something went wrong.
Marking as stale. No activity in 60 days.
No branches or pull requests
How to pre-build the dataset's index ?
I want to avoid using compute node for this task:
The text was updated successfully, but these errors were encountered: