Close runs rows on completion (every run currently stays open forever)

[lwgray]

Close runs rows on completion (every run currently stays open forever)

What is this system, briefly

Marcus is a multi-agent coordinator. When you describe a project, Marcus decomposes it into kanban tasks, spawns AI agents to execute them, and records cost data in a SQLite database (~/.marcus/costs.db). One particular table — runs — represents one invocation of a project (one decomposition + execution pass). The cost-visualization dashboard Cato reads from these tables.

What is the problem

The runs table has lifecycle columns the schema is clearly designed around:

CREATE TABLE runs (
  run_id            TEXT PRIMARY KEY,
  project_id        TEXT NOT NULL,
  started_at        TIMESTAMP NOT NULL,
  ended_at          TIMESTAMP,           -- never written
  total_tasks       INTEGER,             -- never written
  completed_tasks   INTEGER,             -- never written
  blocked_tasks     INTEGER,             -- never written
  num_agents        INTEGER,             -- never written
  ...
);

The only Python code that writes to this table is src/marcus_mcp/tools/nlp.py:230 inside create_project:

state.cost_store.record_run(
    Run(
        run_id=_run_id,
        project_id=_canonical,
        project_name=project_name,
        started_at=_started_at,
        path=_path,
    )
)

That's it. It writes the open state. Nothing ever calls record_run a second time with the close-state fields populated. The UPSERT SQL supports it (ON CONFLICT(run_id) DO UPDATE SET ended_at=excluded.ended_at, ...) — the infrastructure is wired — there just is no caller.

Confirmed against the live database

SELECT COUNT(*), SUM(ended_at IS NULL) FROM runs;
-- 1, 1

The validate-escape-hatch run is the only row in the table, and it has been open since creation. Every row in runs has been open since the table was created.

Why this matters

Three concrete consequences:

1. The dashboard can't filter to "completed runs." Any query that says WHERE ended_at IS NOT NULL returns zero rows. Historical-runs panels show nothing.
2. No wall-clock denominator. The coordination-tax metric (and any future cost-per-second / latency analysis) needs (ended_at - started_at) to normalize spend over time. Without it, you can compute total cost but not cost rate.
3. The audit infrastructure shipped in PR #528 doesn't catch this. That audit verifies "every token is attributed" but doesn't check "every run is closed." A complete data-quality audit should flag stale-open runs.

This is a structural gap that has been around since PR #522 (the experiments → runs rename). The original experiments table likely had the same gap; the rename inherited it.

What needs to change

1. New CostStore.close_run method

Wraps the existing UPSERT with a close-state shape:

def close_run(
    self,
    run_id: str,
    *,
    ended_at: Optional[datetime] = None,
    total_tasks: Optional[int] = None,
    completed_tasks: Optional[int] = None,
    blocked_tasks: Optional[int] = None,
    num_agents: Optional[int] = None,
) -> bool:
    """Mark a run as closed and stamp final stats.

    When ``ended_at`` is None, falls back to ``MAX(timestamp) FROM
    token_events WHERE run_id = ?`` — i.e. the timestamp of the last
    LLM call attributed to this run. That's the best on-disk
    approximation for unattended close (direct-MCP usage where the
    user never calls end_experiment).

    Returns True if a row was updated, False if the run_id was
    unknown.
    """

2. Wire close into end_experiment

The existing MCP tool end_experiment (src/marcus_mcp/tools/experiments.py:190) is the natural close hook for path=marcus and path=posidonius runs — it's called by the runner when work is done. Add a close_run call after monitor.stop() succeeds, using the final task counts from monitor.final_metrics.

3. Backfill script for existing open runs

A one-shot scripts/backfill_run_close_state.py that walks every runs row where ended_at IS NULL and:

• Sets ended_at = MAX(timestamp) FROM token_events WHERE run_id = ?
• Skips rows with zero token_events (truly empty runs — leave open, separate concern)
• Idempotent and re-runnable.

4. Extend the audit to surface open runs

CostAggregator.run_audit and project_audit already report token attribution. Add runs_total and runs_open to the audit dict so the dashboard can show "3 runs closed, 1 still open (last activity 5 minutes ago)". This is the missing data-quality signal.

5. Document the direct-MCP gap

path=direct runs have no explicit close signal — the user manually creates a project, manually invokes agents, manually stops. The backfill script handles those at audit time, but going forward there's no real-time close. Document this limitation in the dashboard footer: "Runs created via direct MCP close automatically when no LLM activity is recorded for 30+ minutes." Or accept the gap and rely on periodic backfill.

Worked example

Today, after a snake-game completes:

SELECT run_id, started_at, ended_at, total_tasks, completed_tasks
FROM runs WHERE project_id = 'snake_proj';
-- run_xyz | 2026-05-13T10:00:00Z | NULL | NULL | NULL

After this issue lands, the same query for a project run via /marcus:

-- run_xyz | 2026-05-13T10:00:00Z | 2026-05-13T10:24:53Z | 14 | 13

14 total tasks → 13 completed → 1 blocked or skipped. Wall-clock time: ~25 minutes. Coordination tax now computable as (total_cost_usd) / ((ended_at - started_at).seconds / 60) per minute.

How to verify it works

1. Run a /marcus experiment end-to-end with this change. Confirm the run's row has ended_at, total_tasks, completed_tasks populated after end_experiment fires.
2. Run the backfill script against the user's existing costs.db:

   $ python scripts/backfill_run_close_state.py
   Backfilled ended_at on N runs from token_events.timestamp
   

3. Open the dashboard. "Tokens by task" and "Cost by agent" still work; new "Run duration" stat appears.
4. Audit dict now includes runs_open — a healthy backfill brings this to 0 for any project whose last event is older than 30 minutes.

Where to look in the code first

File	Purpose
src/cost_tracking/cost_store.py	Add close_run method. Schema unchanged — the columns exist.
src/cost_tracking/cost_aggregator.py	Add runs_open / runs_total to run_audit / project_audit dicts.
src/marcus_mcp/tools/experiments.py	Wire close_run call into end_experiment after monitor.stop().
scripts/backfill_run_close_state.py (new)	One-shot historical fixup.
~/dev/cato/dashboard/src/components/RealTimeTab.tsx	Optional: surface run duration when ended_at is present. Out of scope here.

Glossary

Term	Meaning
runs	SQLite table in costs.db — one row per project invocation.
Close	Stamp the run with ended_at + final task counts so it's no longer considered active.
Backfill	One-shot script that fills missing data in historical rows.
end_experiment	Existing MCP tool the /marcus and Posidonius runners call when work is done.
path=direct	Run created via plain MCP (no automated runner) — no natural close signal.

Out of scope (separate issues if needed)

• Automatic close via timer (background job that closes runs inactive for N minutes). The backfill script + audit signal cover the same gap with simpler ops.
• Reopening a closed run when new events arrive late. Edge case; document but don't build.

Related

• PR #522 — experiments → runs rename (introduced the current open-only write path)
• PR #528 — audit infrastructure (missed the open-run check)
• Issue #527 — broader token audit framework
• Issue #529 — comparative experiments (depends on wall-clock denominator)