v0.7: equivalence_tests + cve_patches - #10
Merged
Conversation
Closes #9. Adds two new pipelines and bumps 0.6.0 -> 0.7.0. equivalence_tests (R2E-style, ICML '24): - Extract module-level Python functions from the target repo using AST filters (LOC range, has args + explicit return, no obvious side effects, exclude dunder/test/main/private names) - Single LLM call generates a pytest test importing BOTH `<name>` and `reference_<name>` from `task_module`, asserting equality across ≥5 diverse inputs - Two-stage in-sandbox verification: stub-must-fail, oracle-must-pass - Emit a Harbor task whose gold patch adds task_module.py (with reference_<name> and <name> set to the original) + a test_r2e_<hash>.py - Different from code_instruct: seeds are real functions, not LLM-invented problems; the LLM only writes the test - Harbor-verified Mean reward 1.000 on pallets/click cve_patches (PatchSeeker/CVE-Bench-inspired): - OSV public API client (no auth, stdlib-only) — POST /v1/query - For each vuln, scan references[] for github.com/<owner>/<repo>/commit/<sha> - Fetch commit diff via `gh api`, split into source/test patches - When test_patch present: full F2P/P2P validation (reuses pr_runtime harness) - When test_patch absent: emit anyway with validation_status="no_test_patch" (still useful as training data; verifier signal is weaker) - Emit Harbor task with CVE description as instruction, fix diff as oracle - Severity filter (low/medium/moderate/high/critical) + ecosystem auto-detect - Harbor-verified Mean reward 1.000 on pallets/werkzeug (CVE-2023-25577) 339 unit tests pass; ruff lint + format clean. Plans: - plans/equivalence_tests_plan.md - (cve_patches: in-flight extension to v0.7; plan was scoped down to MVP) Files: - src/repo2rlenv/osv.py (105 LOC) — OSV API client + severity helpers - src/repo2rlenv/pipelines/_function_extractor.py (216 LOC) — R2E-style filters - src/repo2rlenv/pipelines/equivalence_tests.py (395 LOC) — pipeline class - src/repo2rlenv/pipelines/cve_patches.py (294 LOC) — pipeline class - src/repo2rlenv/github.py: fetch_commit_diff + fetch_commit_parent helpers Out of scope (v0.8): - LLM-judged QA gate (SWE-Bench++ four-layer recipe) - Iterative refinement loop for equivalence_tests - LLM-synthesized PoC for cve_patches (no public PoC + no test_patch case) - Polyglot mutation (Java/JS/Go via tree-sitter)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #9.
Summary
Two new pipelines + bump
0.6.0 → 0.7.0:equivalence_tests(R2E-style function-level synthesis, ICML '24). Extract a real Python function from the repo as a frozen oracle (reference_<name>), LLM writes equivalence tests comparing it against a<name>candidate, gold patch fills<name>with the original implementation. The LLM only writes the test — we already have the ground truth. Lower-variance thancode_instruct; yield scales with the number of qualifying functions.cve_patches(PatchSeeker / CVE-Bench-inspired). OSV public API client (POST /v1/query, no auth needed) → scanreferences[]forgithub.com/<owner>/<repo>/commit/<sha>→ fetch the fix commit's diff → reusepr_runtime's F2P/P2P validation harness whentest_patchis present, emit-only withvalidation_status="no_test_patch"otherwise.Harbor end-to-end
equivalence_testspallets__click-eqv-a2804cc5(getchar)cve_patchespallets__werkzeug-cve-CVE-2023-25577(DoS via multipart)CI
New files
src/repo2rlenv/osv.py— OSV API client + severity helpers (stdlib only)src/repo2rlenv/pipelines/_function_extractor.py— R2E-style AST filterssrc/repo2rlenv/pipelines/equivalence_tests.py— pipeline classsrc/repo2rlenv/pipelines/cve_patches.py— pipeline classtest_osv.py,test_function_extractor.py,test_pipeline_{equivalence_tests,cve_patches}.py(~80 new tests)Notable design decisions
task_module.pyships bothreference_<name>(frozen, used by tests) and<name>(stubbed in env, filled by patch). LLM-written tests assert equality. Catches "tautological test" cheats via the Stage-A invariant (test must FAIL when<name>is stubbed).references[], so we don't need a PatchSeeker-style LLM mapper for the common case. No auth required.pr_runtimehelpers (split_patch_and_test_patch,build_environment_dockerfile,build_eval_script,validate_pr).validation_status="no_test_patch". The verifier signal is weak (just "suite passes with fix applied") but the data is useful as training pairs.Out of scope (v0.8)
equivalence_tests(R2E's "feedback → fix_error" cycle)cve_patches(security implications gating needed)mutation_bugs(Java/JS/Go via tree-sitter)pr_stream🤖 Generated with Claude Code