Skip to content

v1.6.1 — fix the measurement 1.6.0 shipped

Choose a tag to compare

@qazbnm456 qazbnm456 released this 28 Aug 17:43
· 39 commits to main since this release

Two correctness fixes in shipped code. No new public name, no new payload field, no schema change.

A turn's timestamp could come from an earlier turn — and it reached a rendered UI

dspy wraps parse with with_callbacks once per class that defines it, and this kit's own
runtime._LenientJSONAdapter.parse calls super().parse(...). So under the default adapter
(config.adapter == "json") every root turn fired task._MainStepTimer twice with identical
outputs, where the stock JSONAdapter fires once. trace.record_main_trajectory matches a turn to
its live stamp by reasoning, so the surplus stamp was claimable by any later turn repeating
that string — which a retry loop does.

Measured across 85 real traces: all 12 with a ts inversion had a duplicated reasoning, none of
the 58 with unique reasoning did, and 2.1% of per-turn deltas came out negative. A consumer
rendered one as a −338.7s turn duration.

The turn before an inversion is skewed by the same stamp but looks plausible, so it hides. A
consumer can suppress the nonsense value; it can never find the plausible-but-wrong one. That is
why this is fixed where the stamp is made.

_MainStepTimer now stages the outermost parse only, via a per-thread depth from the public
on_adapter_parse_start/on_adapter_parse_end pair. If a future dspy stops firing the start hook,
behaviour degrades to exactly what it was before — never to staging nothing.

A matcher-side fix was tried and rejected. A forward-only cursor repairs non-adjacent
duplicates (27 of 32 in the corpus) but cannot repair adjacent ones (5 of 32): there it only stops
the delta going negative while the stamp stays ~0.1s wrong, trading a loud failure for a silent
one. It ships anyway as defence in depth — _match_ts and _match_exec both scan forward only
now — but it is not the fix.

verify_quote refused correct citations at non-word junctions

Whitespace runs were joined with \s+ uniformly, so a quote that reflowed a line break beside a
delimiter failed even though the citation was exact. The joiner is junction-aware now: \s+
between two word characters, \s* elsewhere. foo bar still cannot verify against foobar — the
false-positive direction stays closed, because an invented claim passing is worse than a real one
refused. Word-ness uses Python's Unicode \w, so 你好 世界 keeps requiring its space; an
ASCII class would have silently started accepting it.

Found by inspection, not by a failure: across ~479 real citations the old and new rules never
disagreed.

Two observable changes, both schema-legal

  • exec_duration_s can be absent where it was present, and a ts can fall back to flush time
    where it was a live stamp. Both fields are optional in trace/v1; a consumer tracking coverage
    will see the number move.
  • verify_quote can report an earlier occurrence — a . b against a.b ... a . b reported
    char 8 and now reports char 0. The line number is often unchanged, so a line-level check will
    not see it.

Docs

New "Reading a trace — the ordering rules" section, written because a downstream consumer read
these traces, concluded "sort by ts", and reordered its turns. main_step events are written in
one block after the run (so a tool_call precedes them in file order while being chronologically
later — 70 of 76 traces); payload["turn"] is authoritative and file order already matches it
(72 of 72); ts places turns against tool calls and nothing else.

Also corrects the claim that every local tool is sub-millisecond: make_grep_files_tool ships
per_match_timeout_s=1.0 and max_total_time_s=30.0. It stays untimed on a measurement
instead — n=146, median 0.029s, max 0.746s, not one call over a second — with the two caveats that
would reopen it now written beside it.

Full changelog: v1.6.0...v1.6.1