shipit-watcher 1.3.0 — datasets

Turn real traffic into a regression suite.

pip install -U shipit-watcher

Capture as it happens

with wt.trace("agent.turn", user_id=user.email) as ctx:
    ...
    if user_reported_it_wrong:
        wt.capture(input=question, metadata={"reported_by": user.email})

Called inside a trace, the origin fills itself in — the row links back to the trace that produced it. The dataset name defaults to WATCHER_DATASET, and the dataset is created on first use.

Replay and compare

wt.run_experiment("regressions", task=..., run_name="prompt-v7", evaluators=[...])

Each item gets its own trace linked under run_name, with evaluator scores attached. The difference between "the new prompt feels better" and "0.82 against 0.71 on the same 40 cases".

An item whose task raises is recorded and the run continues — aborting would discard results already gathered, and a task that fails on one input is itself a finding.

New

• wt.capture, wt.add_item, wt.get_items, wt.create_dataset, wt.run_experiment
• WATCHER_DATASET config

288 tests.