Repository navigation
README
This fixed-case evaluation fetches seven public projects at exact commits, runs Archify's normal CLI from both a Git checkout and a the committed archify.zip from the same revision, grades frozen units and compares outputs. It does not run the target projects. The analyzer is executed; npm ci --ignore-scripts in each archify/modules/code-analysis directory installs its locked dependencies with package lifecycle scripts disabled. Network access to GitHub and npm is required.
Install Git, Node 22.23.2, npm and Python 3.13 yourself and put them on PATH. The recorded run used Python 3.13.16. The harness refuses other Node versions or Python minor versions. It does not install runtimes. Allow disk space for 12 Git checkouts and analysis outputs. Exact commit fetches require GitHub to serve the pinned objects.
python3.13 replay.py --revision python --work /path/to/new-replay --prepare-only
python3.13 replay.py --revision python --work /path/to/new-replay--work must not already exist; each full run starts fresh. Options are labs, combined, python, and paired, pinned in the script. The paired revision is published in Draft #1243 and was fetched by the verified replay. Its committed ZIP digest is also pinned; both modes must meet the frozen paired contract: Redux, Commander, HTTPX and Pluggy ready with exit 0; SLF4J exit 1/incomplete with precisely one known parse-error file per side, available selected definition coverage and both partial-coverage reasons. All eight selected changed definitions and four outgoing imports must score correct. Use --require-paired-contract to apply that gate to another revision. The script checks source commits, tracked cleanliness and selected source hashes before and after analysis, validates bundled inputs, then writes preflight, CLI output, captures, unit grades and a source-versus-archive comparison under the work directory. A failed preflight, missing file, differing result or missing expected comparison output exits nonzero. Do not treat a partial directory as a completed run.
Comparison checks all 27 command arguments, exits and receipt statuses, and requires 12 raw fact files to match byte-for-byte. Ten saved-question and five paired results must match after replacing only rawFactsPath and absolute workflow paths. For older versions, a missing result is accepted only when both sides return the same explicit unsupported-option usage error or, for the three specified Python/Java paired entries, the matching JS/TS-only entry rejection. A pass is equality between packaging modes, not proof of correctness.
provenance.json records SHA-256 hashes of the original private inputs and the public copies. Private filesystem paths and references were replaced by source placeholders or omission markers; frozen expected semantic values and source-file digests were retained. At runtime only declared source-root paths are relocated. The original oracles were not overwritten. Supplemental and outgoing oracle copies retain their original byte hashes. The original freeze program and private reference documents are not distributed; their hashes provide provenance, not independent public access to those documents.
The source-first atomic units were enumerated after prior analyzer exposure; this is not a blind benchmark. Supplemental definition spans and selected outgoing imports were frozen before paired candidate verification, also after previous product studies. Original questions had earlier source-first answers, but their private evidence is not included here. Results cover only selected units and do not establish completeness or performance on other repositories. Inspect unit-results.json and raw evidence, including omissions and explicit unknowns, rather than treating aggregate counts as general accuracy.
Input, syntax, prerequisite and adapted adversarial grader checks passed. The completed fresh-fetch replay below also exercised the graders against the publicly fetched source checkouts.
SLF4J’s known parse-error file is slf4j-migrator/jcl/jclcontent.java, line 19, column 1, on both sides. It is an example intended for conversion, with top-level assignment statements. Its failure makes the full capture partial even though the selected entry definitions are available. The gate preserves that limitation and rejects other partial-coverage causes.
A fresh GitHub-fetch replay of paired candidate 92606d7e8d1a55a800c0ee0ea34753630682c0f5 completed successfully with Node 22.23.2 / Python 3.13.16 on macOS. See receipts/public-replay-92606.json, receipts/unit-results.json and receipts/source-vs-archive.json. All 27 commands matched across source/committed ZIP; 12 raw facts matched byte-for-byte; 10 saved and 5 paired outputs matched. Four paired workflows are ready; SLF4J is intentionally partial for the documented conversion example.