We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.
Already on GitHub? Sign in to your account
In my code, I am enabling a tqdm bar per worker with:
tqdm
global_rank = int(os.environ["DATA_OPTIMIZER_GLOBAL_RANK"]) num_workers = int(os.environ["DATA_OPTIMIZER_NUM_WORKERS"]) local_rank = global_rank % num_workers for example in tqdm(data, position=local_rank): tokens = tokenizer.encode(example) yield tokens
But litdata prints this in each rank:
Rank 3 inferred the following `['no_header_tensor:16']` data format.
Breaking the tqdm bars at the beginning.
Since this print doesn't seem very useful for users, I would suggest that it is removed or put under fast_dev_run or a similar verbose-like flag.
fast_dev_run
The text was updated successfully, but these errors were encountered:
Hi! thanks for your contribution!, great first issue!
Sorry, something went wrong.
No branches or pull requests
馃悰 Bug
In my code, I am enabling a
tqdm
bar per worker with:But litdata prints this in each rank:
Breaking the
tqdm
bars at the beginning.Since this print doesn't seem very useful for users, I would suggest that it is removed or put under
fast_dev_run
or a similar verbose-like flag.The text was updated successfully, but these errors were encountered: