Cleaned evaluation data for SciCode-Verified — a human-verified re-release of the SciCode benchmark with per-problem data defects (signature mismatches, ambiguous units/conventions, non-deterministic tests, degenerate/broken frozen targets) reviewed and corrected under a verify-with-human release gate.
Assets
| File | What it is |
|---|---|
test_data_cleaned.h5 |
The evaluation frozen-target store (--h5py-file). ~1.06 GB. |
problems_test.jsonl |
The 64 released problems (prompts, sub-steps, headers, test cases). |
manifest.json |
Version + md5 of every released file — the integrity anchor. |
Integrity
manifest.json records the expected md5 of each file. Verify after download:
md5sum test_data_cleaned.h5 # must equal manifest.json .h5_md5:
# 2b41a7df40ddc23ce651ec05b8ecb6f8
The evaluation harness re-checks these md5s at startup and refuses to run on a mismatch, so you can never silently score against stale data.
Use
Place the files where the eval expects them and run the harness:
test_data_cleaned.h5 -> scicode_verified/test_data_cleaned.h5
problems_test.jsonl -> scicode_verified/problems_test.jsonl
manifest.json -> scicode_verified/manifest.json
The prebuilt test_data_cleaned.h5 here is the primary path — no need to download the original SciCode h5 or rebuild. (The reproducible recipe is still in the repo: eval_clean/build_clean_h5.py rebuilds this file from the original SciCode test_data.h5 + scicode_verified/targets/, and must reproduce the same md5.)
Attribution
Derived from the SciCode benchmark (https://github.com/scicode-bench/SciCode), licensed under Apache License 2.0. This cleaned data is redistributed under the same Apache-2.0 license; corrections and verification by the SciCode-Verified authors.