Problem
The session-schema adapter reads meta_raw.get("rework_cycles", 0) but SODA's meta.json never writes a rework_cycles field. SODA records rework as generation > 1 on per-phase metadata in meta.json.
Out of 30 SODA sessions in .soda/, 7 have implement.generation=2 — all are reported as rework_cycles=0.
This directly corrupts two key metrics:
first_pass_success_rate — reports 1.0 when it should be ~0.77
self_correction_rate — reports N/A instead of computing actual resolution rates
Root cause
src/raki/adapters/session_schema.py line ~100:
meta_raw.get("rework_cycles", 0)
SODA meta.json stores rework as phases.<phase>.generation (1 = first pass, 2+ = rework). Example from ticket 131:
{
"phases": {
"implement": {
"status": "completed",
"generation": 2,
...
}
}
}
Suggested fix
When rework_cycles is absent from meta.json, compute it from phase metadata:
rework_cycles = max(phase.get("generation", 1) for phase in phases.values()) - 1
Impact
All SODA sessions evaluated with raki run since v0.9.0 have incorrect first_pass_success_rate and self_correction_rate values.
Related
Acceptance Criteria
Task 1: Write failing tests for generation-based rework computation
Files: tests/test_adapters.py
- Write failing test: meta.json without
rework_cycles but with phases.implement.generation=2 → rework_cycles should be 1
- Write failing test: meta.json without
rework_cycles and all phases generation=1 → rework_cycles should be 0
- Write failing test: meta.json with explicit
rework_cycles=3 (backward compat) → should use the explicit value regardless of generation fields
Task 2: Implement generation-based rework computation
Files: src/raki/adapters/session_schema.py
- In the meta loading logic, after reading
meta_raw:
- If
rework_cycles is present in meta_raw, use it (backward compat)
- Otherwise, look for
phases dict in meta_raw, compute max(phase.get("generation", 1) for phase in phases.values()) - 1
- Default to 0 if no phases found
- Verify all 3 failing tests now pass
Task 3: Verification
uv run pytest tests/ -v -m "not slow" — all tests pass
uv run ruff check src/ tests/ && uv run ruff format src/ tests/
uv run ty check src/raki/
Problem
The session-schema adapter reads
meta_raw.get("rework_cycles", 0)but SODA'smeta.jsonnever writes arework_cyclesfield. SODA records rework asgeneration > 1on per-phase metadata inmeta.json.Out of 30 SODA sessions in
.soda/, 7 haveimplement.generation=2— all are reported asrework_cycles=0.This directly corrupts two key metrics:
first_pass_success_rate— reports 1.0 when it should be ~0.77self_correction_rate— reports N/A instead of computing actual resolution ratesRoot cause
src/raki/adapters/session_schema.pyline ~100:SODA
meta.jsonstores rework asphases.<phase>.generation(1 = first pass, 2+ = rework). Example from ticket 131:{ "phases": { "implement": { "status": "completed", "generation": 2, ... } } }Suggested fix
When
rework_cyclesis absent frommeta.json, compute it from phase metadata:Impact
All SODA sessions evaluated with
raki runsince v0.9.0 have incorrectfirst_pass_success_rateandself_correction_ratevalues.Related
Acceptance Criteria
rework_cyclesis absent frommeta.json, adapter computes it frommax(generation) - 1across all phasesrework_cyclesis present, it takes precedence (backward compatibility)generation=2reportrework_cycles=1first_pass_success_ratereflects actual first-pass failuresrework_cyclesfield and computed fromgenerationTask 1: Write failing tests for generation-based rework computation
Files:
tests/test_adapters.pyrework_cyclesbut withphases.implement.generation=2→rework_cyclesshould be 1rework_cyclesand all phasesgeneration=1→rework_cyclesshould be 0rework_cycles=3(backward compat) → should use the explicit value regardless of generation fieldsTask 2: Implement generation-based rework computation
Files:
src/raki/adapters/session_schema.pymeta_raw:rework_cyclesis present inmeta_raw, use it (backward compat)phasesdict inmeta_raw, computemax(phase.get("generation", 1) for phase in phases.values()) - 1Task 3: Verification
uv run pytest tests/ -v -m "not slow"— all tests passuv run ruff check src/ tests/ && uv run ruff format src/ tests/uv run ty check src/raki/