Skip to content

Resolve automatic label name detection when single label provided#40438

Open
ndonthi wants to merge 4 commits intohuggingface:mainfrom
ndonthi:label-names-default
Open

Resolve automatic label name detection when single label provided#40438
ndonthi wants to merge 4 commits intohuggingface:mainfrom
ndonthi:label-names-default

Conversation

@ndonthi
Copy link
Copy Markdown

@ndonthi ndonthi commented Aug 25, 2025

What does this PR do?

Users frequently encounter a confusing KeyError when setting label_names=['label'] in TrainingArguments, leading to errors like:

KeyError: "The `metric_for_best_model` training argument is set to 'eval_f1', which is not found in the evaluation metrics. The available evaluation metrics are: []."

The label_names parameter is designed for advanced multi-label scenarios, but users often mistakenly use label_names=['label'] thinking they need to specify their label column. This breaks the automatic "label" → "labels" conversion that happens during data collation which causes loss calculation to fail silently.

This PR introduces validation for this edge case and fixes the issue in the Trainer to use automatic label detection. If it's preferred, I can make this change where the automatic label detection happens as well!

Implements a code fix to #28530 #40217 and #32373 on huggingface forum

Before submitting

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
with @
@zach-huggingface, @muellerzr @adrienchaton

@Rocketknight1
Copy link
Copy Markdown
Member

cc @SunMarc

@ndonthi
Copy link
Copy Markdown
Author

ndonthi commented Dec 28, 2025

checking in on this! @SunMarc @Rocketknight1

let me know if there's another preferred solution :)

Copy link
Copy Markdown
Member

@SunMarc SunMarc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for digging ! Sorry for the late review, I missed the notification ... Can you add a test also ? One question I have is how did you get the error in the issue ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants