Failure-Set Optimization

Research scaffold for learning from all-wrong rollout groups without inventing relative preferences between incorrect trajectories.

The repository is intentionally strict. A record that cannot be parsed or verified is not treated as an incorrect sample. Multiple final answers are never resolved by silently taking the last \boxed{...} expression.

What is implemented

• strict, versioned JSONL rollout schema;
• collision-resistant trajectory IDs;
• brace-aware \boxed{...} extraction with explicit failure states;
• conservative answer canonicalization and exact verification;
• grouping into mixed, all-correct, all-wrong, and invalid groups;
• answer-equivalence clustering and configurable class weights;
• reference KL projection over answer classes;
• audit gates, provenance manifests, and a synthetic dry-run;
• regression tests for previously observed experimental failure modes.

T1 v2: Real-data converter for HuggingFaceH4/MATH rollouts from the grpo_peer_suppression experiment. One-command reproduction of the full conversion pipeline including golden prompt verification, seed lineage recovery, and complete audit trail.

T2: Versioned symbolic verifier adapter using SymPy for mathematical equivalence checking (fractions, radicals, equations, sets, intervals). Configurable via [verifier].backend = "symbolic" with explicit timeout and audit counters. 89 tests (65 adversarial + 24 acceptance). Subprocess isolation, whitelist input validation, conservative multi-gold aggregation. 100-entry real audit manifest frozen from T1 data. The reference verifier remains unchanged and is the default backend.

T2 acceptance: make t2-acceptance — validates clean Git, dependency lock, SymPy availability, reference regression, T1 immutability. Returns PARTIAL until 100 real human audits are completed.

The model-training integration is deliberately left behind an explicit interface. Do not add a large trainer until the data and objective contracts pass all tests.

Quick start

Python 3.11+ with the dependencies from requirements-lock.txt.

# Run synthetic-only checks (no real checkpoint needed)
make check PYTHON=python3

# Run T1 unit tests (no real checkpoint needed)
make t1-test PYTHON=python3

T1 v2 one-command reproduction

Environment requirements

• Python 3.12+
• Dependencies pinned in requirements-lock.txt: transformers==5.13.1, tokenizers==0.22.2, jinja2==3.1.6, safetensors==0.8.0
• Local Qwen3-1.7B checkpoint at /root/autodl-tmp/models/Qwen/Qwen3-1.7B
• Source rollout data at /root/autodl-tmp/grpo_peer_suppression/step2_k8_results/

Golden prompt verification

make t1-golden PYTHON=python3

Uses the real tokenizer to build the frozen golden prompt and compares its SHA-256 against the independently-generated fixture at tests/fixtures/math_qwen_v1_golden_prompt.json. Fails if the tokenizer is missing, the prompt hash mismatches, or the fixture is corrupted.

Full acceptance

make t1-acceptance PYTHON=python3 T1_OUTPUT_DIR=runs/math_t1_v2_acceptance_001

This single command requires:

• Clean Git worktree (git status --porcelain must be empty)
• All dependencies matching the lock file
• Real checkpoint at the configured path

It runs: unit tests → golden prompt verification → full conversion → artifact integrity checks → acceptance status generation.

Output directory structure

runs/math_t1_v2_acceptance_001/
├── rollouts.jsonl                  # contract-compliant raw records
├── lineage.jsonl                   # per-record provenance
├── conversion_report.json          # conversion audit & quarantine reasons
├── real_data_audit.json            # verifier audit (expected to fail gates)
├── manifest.json                   # full provenance manifest
├── acceptance_status.json          # PASS/FAIL for each dimension
├── golden_prompt_verification.json # golden prompt check result
├── config.toml                     # snapshot of the conversion config
└── exclusions.toml                 # snapshot of the exclusion rules

Output directory is immutable

The output directory must not exist before the run. The runner creates artifacts atomically — all files are written to a temp directory and renamed into place only after all integrity checks pass. If any step fails, no partial output directory is left behind.

Golden prompt

The prompt is reconstructed from the original generation scripts' FEW_SHOT and the Qwen3-1.7B chat template. The SHA-256 is verified against a frozen fixture generated independently by running the original build_prompt with the real tokenizer.

prompt_provenance is "reconstructed_from_original_script_and_tokenizer", never "exact_captured".

Expected real-data audit failure

The require_unique_boxed gate will fail on real data because 98.4% of completions contain multiple \boxed{} expressions (the model outputs both thinking and final answers with boxed notation). This is the expected T1 result. Do not lower the gates.

T2 is complete

T2 (symbolic verifier adapter) is implemented. The SymPy-based backend handles fractions, radicals, equations, sets, and intervals with explicit timeout and audit counters.

T3 is blocked

T3 (answer-span detector) is required to resolve multiple \boxed{} expressions in real data. Until then, real_data_training_ready remains false.

CLI exit code semantics

fspo convert-math returns 0 when and only when:

• T1 data content: PASS
• T1 reproducibility/repository: PASS
• All 9 artifacts are complete
• Audit invariants are self-consistent
• The formal run was published successfully

convert-math returns 0 even when:

• real_data_training_ready = false
• t2 = BLOCKED_AS_EXPECTED
• All groups are INVALID per strict verifier

T2 blocked is expected with real MATH data — it does not indicate a T1 failure.

Exit codes:

Code	Meaning
0	T1 conversion succeeded (all invariants pass, artifacts complete)
1	Parameter error, I/O error, unexpected exception, missing environment
2	T1 content or reproducibility acceptance failed (invariant failure, dirty git, golden mismatch, dependency mismatch)

Source, quarantine, and output groups

The runner tracks three distinct group populations:

Population	Count (current data)	Definition
source_group_count	1411	Groups identified before conversion/exclusion
quarantine_group_count	20	Groups isolated during conversion (ambiguous gold, exclusions, lineage failure)
output_group_count	1391	Groups written to rollouts.jsonl

Invariants:

• source_group_count == quarantine_group_count + output_group_count (1411 = 20 + 1391)
• source_record_count == quarantine_record_count + output_record_count (11288 = 160 + 11128)
• quarantine_problem_keys ∩ output_problem_keys = ∅

Quarantined groups appear in conversion_report.quarantined, never in real_data_audit.invalid_groups.

real_data_audit scope

real_data_audit.json is computed from rollouts.jsonl only — it does not read conversion source categories and does not include quarantine records.

real_data_audit.group_count refers to output groups (1391), not source groups (1411).

real_data_audit.invalid_groups lists only output groups that fail the strict verifier (currently all 1391, pending T2).

Audit invariants

Every real_data_audit.json includes an invariant_checks section with these checks:

Invariant	Description
verification_sum_matches_record_count	CORRECT + INCORRECT + UNVERIFIABLE = record_count
extraction_sum_matches_record_count	OK + MISSING + MULTIPLE + MALFORMED = record_count
group_type_sum_matches_group_count	sum of all group type counts = group_count
record_count_matches_group_count_times_k	record_count = group_count × expected_group_size
quarantine_disjoint_from_output	quarantine source keys ∩ output source keys = ∅
source_partition_matches	source groups = quarantine groups + output groups
source_record_partition_matches	source records = quarantine records + output records
rollouts_lineage_count_matches	rollouts records = lineage records

Any invariant failure blocks the run: runner returns non-zero, no PASS manifest is written, the output directory is not published.

Temp directory lifecycle

The runner creates a single temp directory via tempfile.mkdtemp(prefix=".tmp_fspo_", dir=output_parent). The exact path is held in a variable — no wildcard scanning or blind rm -rf.

Lifecycle:

1. temp_dir is created at pipeline start
2. All artifacts are written inside temp_dir
3. On success: os.rename(temp_dir, output_dir) (atomic publish)
4. On failure: shutil.rmtree(temp_dir) with safety checks (parent directory match, .tmp_fspo_ prefix match)

Never cleaned: successfully published output directories, unknown directories, directories without the .tmp_fspo_ prefix.

Clean Git requirement

make t1-acceptance requires git status --porcelain to be empty. The manifest records the full commit SHA and git_dirty: false. UNCOMMITTED is never allowed as a commit identifier.

Integrated runner tests

# Requires real checkpoint
make t1-runner-test PYTHON=python3

These tests:

• Create a temporary git repository
• Use small source fixtures with frozen/fake-local tokenizer
• Execute the same CLI/runner entry as production (python -m fspo.cli convert-math)
• Generate all 9 artifacts from a single command
• Verify all statistical identities
• Never call internal functions directly

Rollout/lineage hash consistency

Between content-pass runs, the rollouts.jsonl and lineage.jsonl must be byte-identical. Any change to conversion semantics that alters rollout or lineage content causes immediate acceptance failure.

The new acceptance run must compare its rollouts/lineage SHA-256 against the previous content-pass run. Mismatch produces the first differing trajectory_id, field, and reason — and blocks T1 PASS.

make check vs make t1-acceptance

Target	What it validates	Real CKPT?
make check	Synthetic pilot: tests + audit + dry-run	No
make t1-test	All unit tests	No
make t1-runner-test	Integrated runner, audit invariants, temp lifecycle, CLI exit codes	Yes
make t1-golden	Golden prompt hash	Yes
make t1-acceptance	Full T1 v2 pipeline (requires clean git)	Yes

make check passing does NOT imply T1 is complete. Use make t1-acceptance.

Repository layout

configs/                 immutable experiment inputs
configs/math_t1_v2.toml  T1 v2 conversion configuration
docs/                    protocol, data contract, and implementation tasks
examples/                tiny synthetic data only
src/fspo/                audited reference implementation
src/fspo/prompting/      versioned prompt builders
tests/                   regression and invariant tests
tests/fixtures/          frozen golden prompt fixture
runs/                    generated artifacts; ignored by git
requirements-lock.txt    pinned dependency versions

Read AGENTS.md before assigning implementation work to another model. The ordered task list and acceptance criteria are in docs/IMPLEMENTATION_TASKS.md.

Research boundary

The current reference objective operates on answer classes and produces auditable sample weights. It does not claim that a textual normalization is a mathematical proof of equivalence, and it does not yet implement a production PPO/GRPO trainer.