Skip to content

v1.8.0 — the facts a rubric needs, computed once

Choose a tag to compare

@qazbnm456 qazbnm456 released this 30 Aug 13:57
· 24 commits to main since this release

The kit now computes the generic half of a rubric's facts, so a consumer supplies only its domain
half. Three new public names, one optional payload field, nothing removed or re-typed.

The middle stage of a pipeline that was missing

rubric.py has always given you the SHAPE of a rubric — criteria_facts(criteria, facts, lens) is
pure and knows nothing about traces, and the category label stays opaque. What it never gave you
was the facts:

events ──▶ [ compute the facts ] ──▶ [ slice through a lens ] ──▶ CriterionFact
             ↑ everyone hand-wrote this      ↑ already here, already pure

compute_run_facts(events) fills it, emitting exactly RUN_FACT_KEYS — import that tuple rather
than hand-copying names. compute_run_facts_by_run for a file holding several runs.

facts = {**compute_run_facts(load_events(path, run_id)), **my_domain_facts(events)}
per_criterion = criteria_facts(my_criteria, facts, MY_LENS)

RUN_FACT_KEYS is closed and public on purpose: it is what the dict is BUILT against, so a key
shaped like a score cannot reach a consumer's dataset without a diff to a SemVer-governed name.

Two facts nothing computed before

budget_exhausted — did the run stop because its iteration budget ran out? Read from the marker
dspy writes on its own fall-through branch, which the kit has always recorded. No configured cap has
to be staged into the trace, it works on every trace ever written, and it avoids the
main_steps >= cap false positive on a run that submits successfully on its last allowed turn.
Tri-state: None, never False, when the answer is unknown.

fence_refused_turns — turns dspy refused to execute over a markdown fence tag. Named for the
mechanism, because the obvious cause is wrong
, and shipping the wrong name would have been the
expensive part. Across three real corpora, 60 of 60 refusals had the fence buried rather than
leading, and 55 of the 60 are valid Python assigning a documentation page whose text contains a
fenced example — markdown = """# Overview … ```bash … """. dspy's stripper scans the whole cell
including string literals. So the number counts environment friction, not a model failing to follow
a format. One consumer read it the other way and spent two prompt generations suppressing the code
blocks its own pages needed.

Optional snapshot into the trace, OFF by default

TraceRecorder(record_metrics=True) or RLM_TRACE_METRICS=1 folds the facts into
run_end.payload["metrics"] — additive within trace/v1. Computed by re-reading the file just
written and filtered by run_id, so the snapshot is consistent-by-construction with the bytes
beside it, and emitted only when that re-read finds this run's own run_start — an all-zero
dict would be indistinguishable from a measured zero, which is 1.7.0's invariant one layer down.
run_end is recorded from a finally, so a BaseException during the snapshot cannot lose it.

Off by default because it changes what a trace CONTAINS, and because those payloads carry the
escalation prompt. compute_run_facts(events) is the authority in every case; the snapshot is a
convenience for a reader that will not call it.

Unmeasured is not zero

tool_wasted_seconds and tool_total_seconds are None, never 0.0, when nothing carried a
duration — and a measured zero stays 0.0. tool_measured_calls rides alongside because one
measured call in fifty is otherwise indistinguishable from fifty in fifty. Both rates are None on
a run with no turns, and fence_refused_turns: 0 there is unmeasured rather than measured-zero.

Internals worth knowing

_dspy_compat gains three names, and they are deliberately not symmetric. The accepted-fence-tag
set is a real dspy module constant, so its test asserts the introspection path resolves rather
than a value the fallback would satisfy anyway. The forced-final marker is a bare literal with
nothing behind it, so its test drives a real forced-final run instead of asserting the kit against
itself. And the fence decision is a declared verbatim mirror of a _-private dspy parser,
cross-checked against that function itself over a 36-row adversarial table: a regex shortcut was
measured at 1,764 disagreements and 3,855 crashes over 20,016 cells, so every line of the mirror is
load-bearing and individually pinned.

Full changelog: v1.7.0...v1.8.0