Skip to content

bug: session-schema adapter defaults rework_cycles to 0 for SODA sessions #219

@decko

Description

@decko

Problem

The session-schema adapter reads meta_raw.get("rework_cycles", 0) but SODA's meta.json never writes a rework_cycles field. SODA records rework as generation > 1 on per-phase metadata in meta.json.

Out of 30 SODA sessions in .soda/, 7 have implement.generation=2 — all are reported as rework_cycles=0.

This directly corrupts two key metrics:

  • first_pass_success_rate — reports 1.0 when it should be ~0.77
  • self_correction_rate — reports N/A instead of computing actual resolution rates

Root cause

src/raki/adapters/session_schema.py line ~100:

meta_raw.get("rework_cycles", 0)

SODA meta.json stores rework as phases.<phase>.generation (1 = first pass, 2+ = rework). Example from ticket 131:

{
  "phases": {
    "implement": {
      "status": "completed",
      "generation": 2,
      ...
    }
  }
}

Suggested fix

When rework_cycles is absent from meta.json, compute it from phase metadata:

rework_cycles = max(phase.get("generation", 1) for phase in phases.values()) - 1

Impact

All SODA sessions evaluated with raki run since v0.9.0 have incorrect first_pass_success_rate and self_correction_rate values.

Related

Acceptance Criteria

  • When rework_cycles is absent from meta.json, adapter computes it from max(generation) - 1 across all phases
  • When rework_cycles is present, it takes precedence (backward compatibility)
  • SODA sessions with generation=2 report rework_cycles=1
  • first_pass_success_rate reflects actual first-pass failures
  • Tests cover both paths: explicit rework_cycles field and computed from generation

Task 1: Write failing tests for generation-based rework computation

Files: tests/test_adapters.py

  1. Write failing test: meta.json without rework_cycles but with phases.implement.generation=2rework_cycles should be 1
  2. Write failing test: meta.json without rework_cycles and all phases generation=1rework_cycles should be 0
  3. Write failing test: meta.json with explicit rework_cycles=3 (backward compat) → should use the explicit value regardless of generation fields

Task 2: Implement generation-based rework computation

Files: src/raki/adapters/session_schema.py

  1. In the meta loading logic, after reading meta_raw:
    • If rework_cycles is present in meta_raw, use it (backward compat)
    • Otherwise, look for phases dict in meta_raw, compute max(phase.get("generation", 1) for phase in phases.values()) - 1
    • Default to 0 if no phases found
  2. Verify all 3 failing tests now pass

Task 3: Verification

  1. uv run pytest tests/ -v -m "not slow" — all tests pass
  2. uv run ruff check src/ tests/ && uv run ruff format src/ tests/
  3. uv run ty check src/raki/

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions