SciCode-Verified — cleaned evaluation data

Cleaned evaluation data for SciCode-Verified — a human-verified re-release of the SciCode benchmark with per-problem data defects (signature mismatches, ambiguous units/conventions, non-deterministic tests, degenerate/broken frozen targets) reviewed and corrected under a verify-with-human release gate.

Assets

File	What it is
test_data_cleaned.h5	The evaluation frozen-target store (--h5py-file). ~1.06 GB.
problems_test.jsonl	The 64 released problems (prompts, sub-steps, headers, test cases).
manifest.json	Version + md5 of every released file — the integrity anchor.

Integrity

manifest.json records the expected md5 of each file. Verify after download:

md5sum test_data_cleaned.h5     # must equal manifest.json .h5_md5:
                                #   2b41a7df40ddc23ce651ec05b8ecb6f8

The evaluation harness re-checks these md5s at startup and refuses to run on a mismatch, so you can never silently score against stale data.

Use

Place the files where the eval expects them and run the harness:

test_data_cleaned.h5  -> scicode_verified/test_data_cleaned.h5
problems_test.jsonl   -> scicode_verified/problems_test.jsonl
manifest.json         -> scicode_verified/manifest.json

The prebuilt test_data_cleaned.h5 here is the primary path — no need to download the original SciCode h5 or rebuild. (The reproducible recipe is still in the repo: eval_clean/build_clean_h5.py rebuilds this file from the original SciCode test_data.h5 + scicode_verified/targets/, and must reproduce the same md5.)

Attribution

Derived from the SciCode benchmark (https://github.com/scicode-bench/SciCode), licensed under Apache License 2.0. This cleaned data is redistributed under the same Apache-2.0 license; corrections and verification by the SciCode-Verified authors.