run_end records the token budget that was in force and what each attempt actually spent, so a
TRUNCATED completion stops being indistinguishable from a MALFORMED one.
Added
-
run_end.payload.budgets— the generation cap that was APPLIED, per role. A consumer's run
died withAdapterParseError: LM response cannot be serialized to a JSON objecton a response that
opened with the model's own reasoning prose inside the JSON envelope, and it read as a small model
failing to follow the format. It was amax_tokenstruncation. dspy detects that —_check_truncation
testsfinish_reason == "length"— and then onlylogger.warnings it, discarding the datum before
any caller can see it; the only evidence was a container log line that rotates. The exception TYPE
is identical either way, so this is the misdiagnosis 1.4.0 already documented undermax_tokens,
recurring because nothing recorded the one fact that separates them.Recorded alongside the token cap: the ITERATION caps (
max_iterations,max_llm_calls,
max_output_chars) and adroppedflag saying whether_build_rlm'sexcept TypeErrorfired —
that path reverts all three to dspy's own defaults, so without the flag the configured numbers
would read as applied when they were not.max_output_charsmatters to the diagnosis in its own
right: dspy head+tail-caps each REPL output, so "the output was cut off" has THREE independent
mechanisms and a reader has to be able to rule each out.Read off the LM, never from
RLMConfig: an injectedmain_lm/sub_lmis used verbatim, so the
configured cap can be one the call never used — which is exactly the consumer whose run died. The
recordedkeysays which name held it, because dspy rewritesmax_tokensto
max_completion_tokensfor OpenAI reasoning models and a reader of the first name alone gets
Nonefor precisely the thinking-model case this exists to explain. Named keys only — a
trace is a shipped artifact andlm.kwargscarriesapi_keyfor every LM the kit builds. -
run_end.payload.usage— token counts, per ATTEMPT.completion_tokens == capis a
truncation, and unlike a boolean the ratio also shows a turn APPROACHING the cap, which is the
early warning nobody has ever been able to see. Collected through dspy's public usage API, whose
tracker the kit HOLDS — so the counts survive the exception, and the fatal call's tokens are
recorded for a run that raised. That is the whole point: the run being diagnosed is one that died.Per attempt, with
turns_recordedmarking the attempt whose turns are in the trace, because
run_with_retryre-runs the whole trajectory and the attempt that reached the trace is NOT always
the last — a run whose FINAL attempt raises keeps an EARLIER attempt's turns. Scoping usage to
"the attempt with the turns" would have discarded the fatal call's tokens. A run that never
produced a prediction (main_steps: 0, the shape of the incident behind this) records every
attempt with none flagged.Not per TURN, and that is a limit rather than an omission:
sub_modelfalls back to
main_modeland dspy propagates the tracker into its sub-LM workers, so planner turns, sub-LM
escalations and same-model tool-LM calls land in one flat list under one key with no call id and
no timestamp. Nothing in dspy's tracker can separate them. For a distribution over runs use
max(completion_tokens), never the run's SUM — summing many turns against a per-completion cap
answers a cost question, not this one. -
A caller's own
dspy.track_usage()is REUSED, not shadowed. dspy installs a tracker only when
none is set, so installing unconditionally would hand a consumer measuring cost around
task.arun(...)ZERO entries for everything inside — this kit writing a structural zero into
someone else's measurement. The kit reuses an installed tracker and reads a per-attempt SLICE, so
the consumer's own calls are never counted as the run's. One disclosed cost when the kit installs
one because you had none: dspy attaches per-prediction usage only when no tracker is installed, so
adspy.Modulea consumer calls from inside a kit run getsNonefromget_lm_usage(). The
counts are still in the tracker and in the trace; that one accessor stops answering.
Not done, deliberately
-
An in-loop recovery for a truncated or unparseable turn was requested and is refused. dspy DOES
expose the seam —Adapter.__call__/acall, whereJSONAdapteralready re-calls on a parse
failure, and which this kit already subclasses and deliberately strips of that fallback. But a
recovery there is invisible to the trajectory: the RLM loop never sees it, so the corrective
exchange is not amain_step— a second unrecorded turn added to a failure whose defining problem
is that the deciding turn is unrecorded. It also doubles the cost of the runaway actually observed.
The version worth having needs the loop, so it belongs upstream: extending dspy's existing
CodeExecutionErrorfeedback path to a parse failure would make it a real turn with a real number. -
RUN_FACT_KEYSis unchanged. The trace payload gains the fields;compute_run_factsdoes not.
With per-turn attribution impossible, atruncated_turnscount would be per-run and weaker than
its name implies.