The CombinedStreamingDataset example in README does not work

[hiyyg]

🐛 Bug

ValueError: ("When `iterate_over_all` is set to True, the weights argument shouldn't be provided.", ' Instead, it will be computed from the inverse of the dataset length.')

To Reproduce

Code sample

Follow the example in README.

Expected behavior

Environment

• PyTorch Version (e.g., 1.0):
• OS (e.g., Linux):
• How you installed PyTorch (conda, pip, source):
• Build command you used (if compiling from source):
• Python version:
• CUDA/cuDNN version:
• GPU models and configuration:
• Any other relevant information:

Additional context

[deependujha]

CombinedStreamingDataset's init function has iterate_over_all=True as default value.

Can you please try:

combined_dataset = CombinedStreamingDataset(datasets=train_datasets, seed=42, weights=weights, iterate_over_all=False)

And, thanks for pointing out the issue.

[hiyyg]

If iterate_over_all=False, then how is the length of the combined dataset determined? The shortest length among all the sub-datasets? Can we assign a custom combined dataset length?

[deependujha]

From my understanding, the philosophy is: if you have multiple datasets and just want to combine them and then use them, simply use the default case (iterate_over_all = True).

But, if you want the dataset to stream different datasets in some specific ratio (0.25 from first dataset and 0.75 from second dataset), then set iterate_over_all=False, and pass the weight ratio as an argument.

The length of the combined dataset is determined by the first dataset that raise StopIteration error.

I don't think, we have a method to assign custom combined dataset length.

[tchaton]

Hey @hiyyg, yes, the length is an estimation based on the sampling ratio. This was really implemented for pre-training LLM with trillion tokens, so it doesn't fit all use cases.

[tchaton]

However, would you mind making a PR to update the README ?