datasets.utils.info_utils.NonMatchingSplitsSizesError:

[muzi0111]

when I try use 'python main.py opt-1.3b --wbits 4 --abits 4 --eval_ppl --tasks lambada_openai,piqa,arc_easy,arc_challenge,openbookqa,boolq’ to get the result,it return the error as follows:
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='test', num_bytes=1305088, num_examples=4358, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='test', num_bytes=5176698, num_examples=17432, shard_lengths=None, dataset_name='parquet')}, {'expected': SplitInfo(name='train', num_bytes=546500949, num_examples=1801350, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='train', num_bytes=1113622699, num_examples=3676136, shard_lengths=[1650675, 1661350, 364111], dataset_name='parquet')}, {'expected': SplitInfo(name='validation', num_bytes=1159288, num_examples=3760, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='validation', num_bytes=4607450, num_examples=15040, shard_lengths=None, dataset_name='parquet')}]
in all process I only change the dir because of perssion deny,How to solve the error?

[lhoestq]

It seems you are using an old version of datasets, can you try to update datasets and try again ?

I just tried and I'm able to load all these tasks without errors with the latest datasets 2.16.2