v1.7.0 — an escalation nobody recorded is one nobody can count
Every sub-LM escalation is now traced automatically, and the sub-LM wrapper hands dspy back the
response shape dspy handed it. No new public name; trace/v1 gains no event type, envelope
key, or payload field.
An escalation nobody recorded is one nobody can count
CLAUDE.md has always stated that a sub-LM call "is recorded as a sub_call". It was true only
when the consumer remembered to call intercept_sub_lm — a plain dspy.LM is invoked by dspy
directly and recorded nothing.
Surveyed across a fleet of nine consumers, four never wrapped, and two of those had corpora —
141 traces — in which sub_call was identically zero. That is indistinguishable from "measured,
and the model never escalated", and reading it as the second is a mistake this project has already
made: a decision to defer a speculation engine cited the zero as evidence of no escalation. The
real rate, re-derived from what the model actually wrote, was 0.15% — and one of those calls later
proved to be a 235.5-second escalation. An absent event is not a measurement.
RLMTask now wraps a plain sub_lm at the same per-run seam that binds the recorder. Nothing to
change on the consumer side. intercept_sub_lm keeps its real job — pass validators /
postprocessors for a deterministic validate/post-process pipeline — and a consumer with its own
recording wrapper opts out by declaring records_sub_call = True on it.
The bug that made auto-wrapping unsafe until it was fixed
intercept_sub_lm collapsed anything non-list into [outputs]. dspy's RLM._query_lm accepts
two shapes — a typed LMResponse or the legacy list[str | dict] — so a typed response became
[LMResponse] and dspy raised Sub-LM response must contain text, got LMResponse.
Invisible on the default path, reproducible under dspy.context(experimental=True), and on a
clock: dspy's own source says "In DSPy 3.3 and 3.4, ordinary calls preserve the legacy public
return value." The assumption had a two-minor shelf life and no test could see it expire, because
the convention was encoded at the call site instead of in _dspy_compat. It is resolved there now,
with a test that pins dspy's contract so the next change goes red in this repo.
Two rules that shim encodes, both found by adversarial review rather than by a failure:
- an unrecognised shape is handed back untouched so dspy raises its own error — inventing
[""]
converts a loud failure into a silent empty completion that reaches the planner and then the RL
data as a real escalation answer; - substituting text into an
LMResponsedrops the output's later text parts, because
LMOutput.textjoins them — replacing only the first left the rest appended ("AB"round-tripping
to"ABB").
model_as_tool carried the identical defect sixty lines away and is fixed in the same place.
What changes for a trace you already have
- Traces from a consumer that never wrapped gain
sub_callevents, carrying the escalation
prompt (input, truncated to 4,000 chars). A corpus spanning the upgrade is not homogeneous;
run_start.rlm_harnessseparates it. metrics.compute_run_utilization'ssub_calls_totalmoves from an unmeasurable zero to a count,
andexport_actionsgainskind="sub"records — the point of the change, since an RL trainer
doing credit assignment previously could not see an escalation that happened.- A duck-typed sub-LM returning a bare
strnow fails where it used to work; one deriving from
dspy.LMis unaffected.
Full changelog: v1.6.1...v1.7.0