Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

polemo2_in task error with datasets 2.16.1 #1307

Closed
djstrong opened this issue Jan 17, 2024 · 1 comment · Fixed by #1313
Closed

polemo2_in task error with datasets 2.16.1 #1307

djstrong opened this issue Jan 17, 2024 · 1 comment · Fixed by #1313

Comments

@djstrong
Copy link
Contributor

polemo2_in task error with datasets 2.16.1:

Traceback (most recent call last):
  File "~/venv/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "~/lm-evaluation-harness/lm_eval/__main__.py", line 231, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "~/lm-evaluation-harness/lm_eval/utils.py", line 415, in _wrapper
    return fn(*args, **kwargs)
  File "~/lm-evaluation-harness/lm_eval/evaluator.py", line 122, in simple_evaluate
    task_dict = lm_eval.tasks.get_task_dict(tasks)
  File "~/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 275, in get_task_dict
    task_name: get_task(task_name=task_element, config=config),
  File "~/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 217, in get_task
    return TASK_REGISTRY[task_name](config=config)
  File "~/lm-evaluation-harness/lm_eval/api/task.py", line 619, in __init__
    self.download(self.config.dataset_kwargs)
  File "~/lm-evaluation-harness/lm_eval/api/task.py", line 714, in download
    self.dataset = datasets.load_dataset(
  File "~/venv/lib/python3.10/site-packages/datasets/load.py", line 2523, in load_dataset
    builder_instance = load_dataset_builder(
  File "~/venv/lib/python3.10/site-packages/datasets/load.py", line 2232, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "~/venv/lib/python3.10/site-packages/datasets/builder.py", line 371, in __init__
    self.config, self.config_id = self._create_builder_config(
  File "~/venv/lib/python3.10/site-packages/datasets/builder.py", line 592, in _create_builder_config
    raise ValueError(
ValueError: BuilderConfig 'klej-polemo2-in' not found. Available: ['default']

but it works better with datasets 2.15.0.

@lhoestq
Copy link
Contributor

lhoestq commented Jan 18, 2024

Hi ! klej-polemo2-in is not a valid subset for https://huggingface.co/datasets/allegro/klej-polemo2-in

I opened a PR to fix this issue in the yaml task file

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants