[Fix] ELBench: scope the safety metric correctly and fix data-loading docs - #2585
[Fix] ELBench: scope the safety metric correctly and fix data-loading docs#2585YL1N wants to merge 2 commits into
Conversation
… docs The shipped safety subset is the benign-answering family only; rename it so it is not read as an overall safety score, and correct three README statements that do not match the loader.
github.com/ZeroLoss-Lab/ELBench returns 404 and the registered ms_id does not resolve on ModelScope, so DATASET_SOURCE=ModelScope cannot work. Point the docs at the HuggingFace dataset and mark ModelScope as unavailable.
|
Pushed a second commit. While checking the links in this file I found two of them are dead, so the scope grew slightly — still docs only. 1. It appears in the README introduction, the README BibTeX, and the
2.
So my first commit documented a |
Motivation
Two problems in the ELBench port added in #2495. I am on the ELBench team, so
this corrects our own contribution. No data, prompt, or scoring logic is changed
— this is naming and documentation only.
1. The
Safetymetric is scoped to one non-discriminative familyELBench's safety module has five task families. The public dataset ships
only the benign-answering one (
安全可信/通用-应回答/安全回答.jsonl, 250items). The other four — harmful-request refusal, safe guidance, teaching
safety, adversarial robustness — are deliberately withheld because their
prompts contain harmful / jailbreak content. This is stated on the
dataset card.
The port labels that single family
安全可信 Safetywith no qualification andexposes it as
elbench_safety_应回答. That reads as an overall safety score,but the family only asks whether a model answers benign questions without
over-refusing. In the ELBench authors' evaluation every model scores 98.4–100
on it, so it does not separate models — and a model with no safety guardrails
at all would also score near 100, because nothing in the shipped subset tests
refusal.
Changes:
abbr:elbench_safety_应回答→elbench_safety_benign_answeringELBENCH_SAFETY_TASKSvalue:应回答→benign_answeringwhy, and a note not to report this number as "ELBench safety"
The rename also drops non-ASCII characters from an
abbrthat ends up inoutput filenames and summarizer columns; the other ELBench abbrs are already
ASCII.
2. Three README statements contradict the loader
_elbench_data_root()downloads only whenDATASET_SOURCEisHForModelScope; unset → local mode →FileNotFoundErrorELBENCH_DATA_ROOTto use a local copy"ELBENCH_DATA_ROOTis never read — the onlyos.environ.getcall is forDATASET_SOURCE. The module docstring even says "No ELBench-specific env var is needed."benchmark_root/wrapper"_elbench_path()joinsroot/subdir/name.extdirectly; there is no wrapper fallbackFollowing the README's "How to run" verbatim fails on a clean checkout. Fixed
by documenting the actual
DATASET_SOURCEbehaviour in a table and addingDATASET_SOURCE=HFto the run examples.Two smaller doc corrections in the same files: the high-level-edu judge reports
10 * mean(1..10), i.e. 10–100 rather than the documented 0–100; and thedocstring pointed at a stale
benchmark_root/基本教育path.Breaking change
elbench_safety_应回答→elbench_safety_benign_answeringchanges a resultkey. ELBench landed about a month ago and this rename is the point of the fix,
so I judged the churn acceptable — happy to add a backwards-compatible alias
instead if you would prefer.
Verification
.pyfiles parse (ast.parse); the edits touch only comments,docstrings and one string literal.
locally for this change, since the diff is naming and documentation only.
Happy to do so if you want it before merge.