Skip to content

[BUG] Cache management failed for tasks using sample_params to update function attrs #1026

@colinzuo

Description

@colinzuo

Describe the bug

Metric:call used "&".join(f"{k}={v}" for k, v in sample_params.items()) to update metric name, but the value might be function, which will change for each run.

To Reproduce

Task that use sample_params, such as:
leaderboard:gsm8k:0

metrics=[
    Metrics.exact_match(sample_params={"normalize_gold": gsm8k_normalizer, "normalize_pred": gsm8k_normalizer})
],

Expected behavior

Cache should work

Version info

main branch

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions