Skip to content

[BUG] Forced to hit You cannot select the number of dataset splits with litellm #800

@jamesbraza

Description

@jamesbraza

Describe the bug

When using LiteLLMClient we are forced to hit a warning:

[2025-06-06 18:21:13,616] [ WARNING]: You cannot select the number of dataset splits for a generative evaluation at the moment. Automatically inferring. (data.py:237)

To Reproduce

  1. From https://github.com/huggingface/lighteval/blob/v0.10.0/src/lighteval/models/abstract_model.py#L58, DATASET_SPLITS = 4.
  2. From https://github.com/huggingface/lighteval/blob/v0.10.0/src/lighteval/models/litellm_model.py#L229, we pass that to num_dataset_splits.
  3. Then from https://github.com/huggingface/lighteval/blob/v0.10.0/src/lighteval/data.py#L83-L85, we are forced to hit the above warning.

Expected behavior

Not being forced to hit a warning.

Version info

lighteval==0.10.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions