[BUG] FileNotFoundError with newest lighteval 0.11.0

[jxmorris12]

Describe the bug

I receive a FileNotFoundError:

│ /home/jxm/anaconda3/lib/python3.12/site-packages/datasets/load.py:1033 in dataset_module_factory                                                                                                                                             │
│                                                                                                                                                                                                                                              │
│   1030 │   │   │   │   │   ) from None                                                                                                                                                                                                       │
│   1031 │   │   │   │   raise e1 from None                                                                                                                                                                                                    │
│   1032 │   else:                                                                                                                                                                                                                             │
│ ❱ 1033 │   │   raise FileNotFoundError(f"Couldn't find any data file at {relative_to_absolute_p                                                                                                                                              │
│   1034                                                                                                                                                                                                                                       │
│   1035                                                                                                                                                                                                                                       │
│   1036 def load_dataset_builder(                                                                                                                                                                                                             │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
FileNotFoundError: Couldn't find any data file at /home/jxm/lighteval/openai/gsm8k.

To Reproduce

lighteval accelerate "model_name=gpt2" "lighteval|gsm8k|0|0"

Expected behavior

I expect it to work and run evaluation.

Version info

0.11.0 on linux with conda/uv pip

[amstu2]

This has been fixed in #986, but I'm not sure when the next release will be.

I'd recommend installing directly from GitHub for now (e.g. uv add "lighteval @ git+https://github.com/huggingface/lighteval")

[NathanHB]

hey ! yes it's a bug that made it's way to the release. The fix is on main branch and will be patched very soon, thanks for your patience