Skip to content

[Fix] ELBench: scope the safety metric correctly and fix data-loading docs - #2585

Open
YL1N wants to merge 2 commits into
open-compass:mainfrom
YL1N:fix/elbench-safety-scope-and-docs
Open

[Fix] ELBench: scope the safety metric correctly and fix data-loading docs#2585
YL1N wants to merge 2 commits into
open-compass:mainfrom
YL1N:fix/elbench-safety-scope-and-docs

Conversation

@YL1N

@YL1N YL1N commented Aug 4, 2026

Copy link
Copy Markdown

Motivation

Two problems in the ELBench port added in #2495. I am on the ELBench team, so
this corrects our own contribution. No data, prompt, or scoring logic is changed
— this is naming and documentation only.

1. The Safety metric is scoped to one non-discriminative family

ELBench's safety module has five task families. The public dataset ships
only the benign-answering one (安全可信/通用-应回答/安全回答.jsonl, 250
items). The other four — harmful-request refusal, safe guidance, teaching
safety, adversarial robustness — are deliberately withheld because their
prompts contain harmful / jailbreak content. This is stated on the
dataset card.

The port labels that single family 安全可信 Safety with no qualification and
exposes it as elbench_safety_应回答. That reads as an overall safety score,
but the family only asks whether a model answers benign questions without
over-refusing. In the ELBench authors' evaluation every model scores 98.4–100
on it, so it does not separate models — and a model with no safety guardrails
at all would also score near 100, because nothing in the shipped subset tests
refusal.

Changes:

  • abbr: elbench_safety_应回答elbench_safety_benign_answering
  • ELBENCH_SAFETY_TASKS value: 应回答benign_answering
  • README table row + footnote: the five families, which four are withheld and
    why, and a note not to report this number as "ELBench safety"
  • The same note in the module docstring

The rename also drops non-ASCII characters from an abbr that ends up in
output filenames and summarizer columns; the other ELBench abbrs are already
ASCII.

2. Three README statements contradict the loader

README says Loader actually does
data is "downloaded automatically … on first run" _elbench_data_root() downloads only when DATASET_SOURCE is HF or ModelScope; unset → local mode → FileNotFoundError
"set ELBENCH_DATA_ROOT to use a local copy" ELBENCH_DATA_ROOT is never read — the only os.environ.get call is for DATASET_SOURCE. The module docstring even says "No ELBench-specific env var is needed."
"accepts either this root-level layout or a benchmark_root/ wrapper" _elbench_path() joins root/subdir/name.ext directly; there is no wrapper fallback

Following the README's "How to run" verbatim fails on a clean checkout. Fixed
by documenting the actual DATASET_SOURCE behaviour in a table and adding
DATASET_SOURCE=HF to the run examples.

Two smaller doc corrections in the same files: the high-level-edu judge reports
10 * mean(1..10), i.e. 10–100 rather than the documented 0–100; and the
docstring pointed at a stale benchmark_root/基本教育 path.

Breaking change

elbench_safety_应回答elbench_safety_benign_answering changes a result
key. ELBench landed about a month ago and this rename is the point of the fix,
so I judged the churn acceptable — happy to add a backwards-compatible alias
instead if you would prefer.

Verification

  • Both modified .py files parse (ast.parse); the edits touch only comments,
    docstrings and one string literal.
  • I have not run an end-to-end evaluation or OpenCompass's pre-commit hooks
    locally for this change, since the diff is naming and documentation only.
    Happy to do so if you want it before merge.

… docs

The shipped safety subset is the benign-answering family only; rename it so it
is not read as an overall safety score, and correct three README statements
that do not match the loader.
github.com/ZeroLoss-Lab/ELBench returns 404 and the registered ms_id does not
resolve on ModelScope, so DATASET_SOURCE=ModelScope cannot work. Point the docs
at the HuggingFace dataset and mark ModelScope as unavailable.
@YL1N

YL1N commented Aug 5, 2026

Copy link
Copy Markdown
Author

Pushed a second commit. While checking the links in this file I found two of them are dead, so the scope grew slightly — still docs only.

1. https://github.com/ZeroLoss-Lab/ELBench returns 404.

It appears in the README introduction, the README BibTeX, and the elbench.py module docstring — all added by #2495. The ZeroLoss-Lab org exists but has no public ELBench repo. All three now point at the HuggingFace dataset, which resolves:

$ curl -o /dev/null -w '%{http_code}' -L https://github.com/ZeroLoss-Lab/ELBench
404
$ curl -o /dev/null -w '%{http_code}' -L https://huggingface.co/datasets/ZeroLoss-Lab/ELBench
200

dataset-index.yml also has paper: https://github.com/ZeroLoss-Lab/ELBench. I left it alone to keep this PR scoped — happy to fix it here if you'd rather.

2. DATASET_SOURCE=ModelScope cannot work — the dataset is not on ModelScope.

datasets_info.py registers ms_id: ZeroLoss-Lab/ELBench, but that dataset does not exist. The web URL returns 200, which is a soft 404 from the SPA; the API is unambiguous:

$ curl -s https://www.modelscope.cn/api/v1/datasets/ZeroLoss-Lab/ELBench
{"Code":10020101002,"Message":"不存在的数据集",...}

So my first commit documented a ModelScope option that always fails. The table now marks it unavailable and says HuggingFace is the only working source, with the same note in the loader comment. I did not touch the ms_id entry in datasets_info.py — leaving it means the path starts working as soon as the dataset is mirrored, and removing it would be a behaviour change rather than a doc fix. Let me know if you'd prefer it dropped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants