Skip to content

[BUG] TInyBenchmark metrics need conversion from string to Metric #991

@amstu2

Description

@amstu2

Describe the bug

When running lighteval tasks inspect "lighteval|gsm8k|0", an error is encountered: 'str' object has no attribute 'category'. This is because the tinyBenchmark tasks metrics have not been updated to be Metric classes (https://github.com/huggingface/lighteval/blob/main/src/lighteval/tasks/extended/tiny_benchmarks/main.py#L259)

To Reproduce

Run lighteval tasks inspect "lighteval|gsm8k|0" in the command line.

Expected behavior

Command successfully executes and task information for gsm8k is displayed to the user.

Version info

Commit 5137e03 on main

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions