[BUG] Datasets module got an unexpected keyword argument 'trust_remote_code'

[mertbozkir]

I'm getting the following issue when I try to use lighteval with vllm!

.

╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /arf/home/mbozkir/miniforge3/envs/eval/lib/python3.11/site-packages/lighteva │
│ l/main_vllm.py:147 in vllm                                                   │
│                                                                              │
│   144 │   │   metric_options = {}                                            │
│   145 │   │   model_config = VLLMModelConfig.from_args(model_args)           │
│   146 │                                                                      │
│ ❱ 147 │   pipeline = Pipeline(                                               │
│   148 │   │   tasks=tasks,                                                   │
│   149 │   │   pipeline_parameters=pipeline_params,                           │
│   150 │   │   evaluation_tracker=evaluation_tracker,                         │
│                                                                              │
│ /arf/home/mbozkir/miniforge3/envs/eval/lib/python3.11/site-packages/lighteva │
│ l/pipeline.py:162 in __init__                                                │
│                                                                              │
│   159 │   │                                                                  │
│   160 │   │   self.evaluation_tracker.general_config_logger.log_model_info(g │
│   161 │   │   self._init_random_seeds()                                      │
│ ❱ 162 │   │   self._init_tasks_and_requests(tasks=tasks)                     │
│   163 │   │   # Final results                                                │
│   164 │   │   self.final_dict: dict | None = None                            │
│   165                                                                        │
│                                                                              │
│ /arf/home/mbozkir/miniforge3/envs/eval/lib/python3.11/site-packages/lighteva │
│ l/pipeline.py:216 in _init_tasks_and_requests                                │
│                                                                              │
│   213 │   │   │   registry = Registry(                                       │
│   214 │   │   │   │   custom_tasks=self.pipeline_parameters.custom_tasks_dir │
│   215 │   │   │   )                                                          │
│ ❱ 216 │   │   │   task_names_list, fewshots_dict = taskinfo_selector(tasks,  │
│   217 │   │   │   task_dict = registry.get_task_dict(task_names_list)        │
│   218 │   │   │   # If there are metric_options defined from the yaml file,  │
│   219 │   │   │   # review if they have to be updated.                       │
│                                                                              │
│ /arf/home/mbozkir/miniforge3/envs/eval/lib/python3.11/site-packages/lighteva │
│ l/tasks/registry.py:300 in taskinfo_selector                                 │
│                                                                              │
│   297 │   expanded_tasks_list: list[str] = []                                │
│   298 │   for maybe_task_group in tasks_list:                                │
│   299 │   │   # We either expand the group (in case it's a group name), or w │
│ ❱ 300 │   │   expanded_tasks = task_registry.task_groups_dict.get(maybe_task │
│   301 │   │   expanded_tasks_list.extend(expanded_tasks)                     │
│   302 │                                                                      │
│   303 │   for task in expanded_tasks_list:                                   │
│                                                                              │
│ /arf/home/mbozkir/miniforge3/envs/eval/lib/python3.11/site-packages/lighteva │
│ l/tasks/registry.py:191 in task_groups_dict                                  │
│                                                                              │
│   188 │   │   """                                                            │
│   189 │   │   if self._custom_tasks is None:                                 │
│   190 │   │   │   return {}                                                  │
│ ❱ 191 │   │   custom_tasks_module = create_custom_tasks_module(custom_tasks= │
│   192 │   │   tasks_group_dict = {}                                          │
│   193 │   │   if hasattr(custom_tasks_module, "TASKS_GROUPS"):               │
│   194 │   │   │   tasks_group_dict = custom_tasks_module.TASKS_GROUPS        │
│                                                                              │
│ /arf/home/mbozkir/miniforge3/envs/eval/lib/python3.11/site-packages/lighteva │
│ l/tasks/registry.py:259 in create_custom_tasks_module                        │
│                                                                              │
│   256 │   if isinstance(custom_tasks, ModuleType):                           │
│   257 │   │   return custom_tasks                                            │
│   258 │   if isinstance(custom_tasks, (str, Path)) and os.path.exists(custom │
│ ❱ 259 │   │   dataset_module = dataset_module_factory(str(custom_tasks), tru │
│   260 │   │   return importlib.import_module(dataset_module.module_path)     │
│   261 │   if isinstance(custom_tasks, (str, Path)):                          │
│   262 │   │   return importlib.import_module(str(custom_tasks))              │
│                                                                              │
│ /arf/home/mbozkir/miniforge3/envs/eval/lib/python3.11/site-packages/datasets │
│ /load.py:878 in dataset_module_factory                                       │
│                                                                              │
│    875 │   │   DatasetModule                                                 │
│    876 │   """                                                               │
│    877 │   if download_config is None:                                       │
│ ❱  878 │   │   download_config = DownloadConfig(**download_kwargs)           │
│    879 │   download_mode = DownloadMode(download_mode or DownloadMode.REUSE_ │
│    880 │   download_config.extract_compressed_file = True                    │
│    881 │   download_config.force_extract = True                              │
╰──────────────────────────────────────────────────────────────────────────────╯
TypeError: DownloadConfig.__init__() got an unexpected keyword argument 
'trust_remote_code'

my slurm related field.

execute_task() {
    local task_name="$1"
    echo "Starting task: $task_name"
    echo "Time: $(date)"
    
    export RAY_DISABLE_IMPORT_WARNING=1
    export RAY_DEDUP_LOGS=0
    export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
    export VLLM_WORKER_MULTIPROC_METHOD=spawn && lighteval vllm \
        "model_name=/arf/scratch/mbozkir/result-final-model/cpt_Qwen3-1.7B-Base_axolotl,tensor_parallel_size=2,data_parallel_size=2,max_model_length=8192,trust_remote_code=True" \
        "$task_name" \
        --custom-tasks "/arf/home/mbozkir/miniforge3/envs/axo/lib/python3.11/site-packages/lighteval/tasks/multilingual/tasks.py"
    
    # Check if the task completed successfully
    if [ $? -eq 0 ]; then
        echo "Completed task: $task_name successfully"
    else
        echo "Task $task_name failed with exit code $?"
        # Optionally, you can decide whether to continue or exit
        # exit 1  # Uncomment this line to stop on first failure
    fi
    
    echo "Finished task: $task_name at $(date)"
    echo "----------------------------------------"
}

### Execute tasks sequentially
for task in "${task_names[@]}"; do
    execute_task "$task"
done

lighteval 0.10.0
datasets 4.0.0
vllm 0.10.1.1

Security features from the Hugging Face datasets library

cc: @lhoestq

[clefourrier]

Hi! You need to install from main if you want to use datasets >= 4.0.0, last release was on a version prior to 4.0 where trust remote code still existed

[mertbozkir]

I'm still getting this error. Can you please specify the version matrix?

 ❱ 244 │   │   registry = Registry(tasks=tasks, custom_tasks=self.pipeline_parameters.custom_ta                                                                                     │
│   245 │   │                                                                                                                                                                        │
│   246 │   │   # load the tasks from the configs and their datasets                                                                                                                 │
│   247 │   │   self.tasks_dict: dict[str, LightevalTask] = registry.load_tasks()                                                                                                    │
│                                                                                                                                                                                    │
│ /arf/home/mbozkir/miniforge3/envs/eval/lib/python3.11/site-packages/lighteval/tasks/registry.py:161 in __init__                                                                    │
│                                                                                                                                                                                    │
│   158 │   │   # We load all task to                                                                                                                                                │
│   159 │   │   self._task_registry = self._load_full_registry()                                                                                                                     │
│   160 │   │                                                                                                                                                                        │
│ ❱ 161 │   │   self.task_to_configs = self._update_task_configs()                                                                                                                   │
│   162 │                                                                                                                                                                            │
│   163 │   def _get_full_task_list_from_input_string(self, tasks: str | Path) -> list[str]:                                                                                         │
│   164 │   │   """Converts an input string (either a path to file with a list of tasks or a str                                                                                     │
│                                                                                                                                                                                    │
│ /arf/home/mbozkir/miniforge3/envs/eval/lib/python3.11/site-packages/lighteval/tasks/registry.py:310 in _update_task_configs                                                        │
│                                                                                                                                                                                    │
│   307 │   │   │   │   # We load each config                                                                                                                                        │
│   308 │   │   │   │   config = self._task_registry.get(expanded_task)                                                                                                              │
│   309 │   │   │   │   if config is None:                                                                                                                                           │
│ ❱ 310 │   │   │   │   │   raise ValueError(f"Cannot find task {expanded_task} in task list or                                                                                      │
│   311 │   │   │   │                                                                                                                                                                │
│   312 │   │   │   │   config = copy.deepcopy(config)                                                                                                                               │
│   313 │   │   │   │   config.num_fewshots = few_shot                                                                                                                               │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Cannot find task lighteval|community_truthfulqa_tur_hybrid:mc1 in task list or in custom task registry
[ble: exit 1][ble: elapsed 11.709s (CPU 207.9%)] ./run_lighteval.sh 

pip list | grep vllm
vllm                              0.10.1.1
pip list | grep lighteval
lighteval                         0.10.1.dev0
pip list | grep datasets
datasets                          4.0.0

Even though it shows up when I use lighteval list tasks --custom-tasks .... | grep "community_truthfulqa" !

[mertbozkir]

custom tasks path - /arf/home/mbozkir/miniforge3/envs/eval/lib/python3.11/site-packages/lighteval/tasks/multilingual/tasks.py

[clefourrier]

Did you do, in a new env, pip install .[multilingual] from source? (you need the multilingual dependencies to use the multilingual tasks)

[mertbozkir]

yes, but i figure it out.

This worked for me:

lighteval vllm \
        "$vllm_config" \
        --custom-tasks "/arf/home/mbozkir/miniforge3/envs/eval/lib/python3.11/site-packages/lighteval/tasks/multilingual/tasks.py" \
        "$task_name" \

the order was -> vllm_config, task_name, custom_task
the right way should be -> vllm_config, custom_task, task_name

[NathanHB]

mhh both should work though, what error do you get in the first case ?