Skip to content

v1.7.0 — an escalation nobody recorded is one nobody can count

Choose a tag to compare

@qazbnm456 qazbnm456 released this 29 Aug 14:33
· 30 commits to main since this release

Every sub-LM escalation is now traced automatically, and the sub-LM wrapper hands dspy back the
response shape dspy handed it. No new public name; trace/v1 gains no event type, envelope
key, or payload field.

An escalation nobody recorded is one nobody can count

CLAUDE.md has always stated that a sub-LM call "is recorded as a sub_call". It was true only
when the consumer remembered to call intercept_sub_lm — a plain dspy.LM is invoked by dspy
directly and recorded nothing.

Surveyed across a fleet of nine consumers, four never wrapped, and two of those had corpora —
141 traces — in which sub_call was identically zero. That is indistinguishable from "measured,
and the model never escalated", and reading it as the second is a mistake this project has already
made: a decision to defer a speculation engine cited the zero as evidence of no escalation. The
real rate, re-derived from what the model actually wrote, was 0.15% — and one of those calls later
proved to be a 235.5-second escalation. An absent event is not a measurement.

RLMTask now wraps a plain sub_lm at the same per-run seam that binds the recorder. Nothing to
change on the consumer side. intercept_sub_lm keeps its real job — pass validators /
postprocessors for a deterministic validate/post-process pipeline — and a consumer with its own
recording wrapper opts out by declaring records_sub_call = True on it.

The bug that made auto-wrapping unsafe until it was fixed

intercept_sub_lm collapsed anything non-list into [outputs]. dspy's RLM._query_lm accepts
two shapes — a typed LMResponse or the legacy list[str | dict] — so a typed response became
[LMResponse] and dspy raised Sub-LM response must contain text, got LMResponse.

Invisible on the default path, reproducible under dspy.context(experimental=True), and on a
clock: dspy's own source says "In DSPy 3.3 and 3.4, ordinary calls preserve the legacy public
return value."
The assumption had a two-minor shelf life and no test could see it expire, because
the convention was encoded at the call site instead of in _dspy_compat. It is resolved there now,
with a test that pins dspy's contract so the next change goes red in this repo.

Two rules that shim encodes, both found by adversarial review rather than by a failure:

  • an unrecognised shape is handed back untouched so dspy raises its own error — inventing [""]
    converts a loud failure into a silent empty completion that reaches the planner and then the RL
    data as a real escalation answer;
  • substituting text into an LMResponse drops the output's later text parts, because
    LMOutput.text joins them — replacing only the first left the rest appended ("AB" round-tripping
    to "ABB").

model_as_tool carried the identical defect sixty lines away and is fixed in the same place.

What changes for a trace you already have

  • Traces from a consumer that never wrapped gain sub_call events, carrying the escalation
    prompt (input, truncated to 4,000 chars). A corpus spanning the upgrade is not homogeneous;
    run_start.rlm_harness separates it.
  • metrics.compute_run_utilization's sub_calls_total moves from an unmeasurable zero to a count,
    and export_actions gains kind="sub" records — the point of the change, since an RL trainer
    doing credit assignment previously could not see an escalation that happened.
  • A duck-typed sub-LM returning a bare str now fails where it used to work; one deriving from
    dspy.LM is unaffected.

Full changelog: v1.6.1...v1.7.0