fix(observe): emit freshness on the wire, not verdict - #2
Merged
Conversation
wireValidation in run/bundle.ts still wrote the old verdict field name, so every browser-written bundle was invalid against the published run-bundle v1 schema (freshness is required) and disagreed with Go-written bundles on a field name. Renames the observe-side type and field throughout (Verdict -> Freshness, ValidationResult.verdict / ValidationOutcome.verdict -> .freshness) to match sdk-go v0.3.0, runtime-go v0.5.0 and cli v0.6.0, and refreshes the vendored schema fixture from the canonical formats/run-bundle/v1/schema.json (it was stale in exactly the same way, which is why the schema-validation test did not already catch this). AssertionVerdict in deterministic/assertions.ts is left untouched: it is a genuinely different concept (an AI-judged assertion outcome), not the runtime's own click-validity check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SkkrXdtf8mU2GRo2hHsHT1 Signed-off-by: Alexander Trakhimenok <alex@trakhimenok.com>
There was a problem hiding this comment.
Pull request overview
This PR aligns the TypeScript runtime’s observe/validation terminology and wire format with the published run-bundle v1 schema by renaming the click-validity concept from verdict to freshness (values remain fresh/stale). This ensures browser-written bundles validate against the schema and match Go-written bundles on the field name.
Changes:
- Renamed observe-side validation types/fields from
Verdict/verdicttoFreshness/freshnessacross the runtime and call sites. - Updated run-bundle wire serialisation to emit
freshnessinstead ofverdict. - Refreshed the vendored run-bundle v1 schema fixture and updated affected tests accordingly.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| src/session/fixtures/run-bundle.v1.schema.json | Updates schema fixture to require/describe freshness rather than verdict. |
| src/run/report.ts | Updates report logic to interpret validation via validation.freshness. |
| src/run/report.test.ts | Adjusts report test data to use freshness. |
| src/run/bundle.ts | Fixes wire serialisation to emit freshness on the wire. |
| src/observe/observe.ts | Renames public observe types/fields (Verdict → Freshness, verdict → freshness) and updates docs. |
| src/observe/observe.test.ts | Updates observe validation tests to assert freshness. |
| src/observe/engine.ts | Updates engine validation result shape to return freshness and updates docs. |
| src/actor/openai/prompt.ts | Updates prompt history rendering to print validation.freshness. |
| src/actor/loop.ts | Updates loop validation plumbing to carry freshness and branch on it. |
| src/actor/loop.test.ts | Updates loop stale-click test to check freshness. |
| src/actor/loop-event.ts | Renames loop event validation field/type usage to Freshness/freshness. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| * Checks `proposal` against the Engine's CURRENT journal state — never | ||
| * against the (possibly outdated) Observation the actor originally saw — | ||
| * and returns a deterministic fresh/stale verdict with a reason. It does | ||
| * and returns a deterministic fresh/stale freshness with a reason. It does |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
wireValidationinsrc/run/bundle.tsstill emitted the oldverdictfield name, so every browser-written run bundle was invalid against the publishedformats/run-bundle/v1/schema.json(freshnessis required there) and disagreed with Go-written bundles on a field name — a principle 7 violation ("one scenario file, two runtimes, same verdict").Verdict->Freshness(valuesfresh/staleunchanged),ValidationResult.verdict->.freshness,ValidationOutcome.verdict->.freshness, and every call site (observe/engine.ts,actor/loop.ts,actor/loop-event.ts,actor/openai/prompt.ts,run/report.ts,run/bundle.ts) and test.src/session/__fixtures__/run-bundle.v1.schema.jsonfrom the canonicalchatwright/chatwrightcopy — confirmed byte-identical after the copy.deterministic/assertions.ts'sAssertionVerdict(pass/fail/unverified) untouched — that one is correctly named, a genuinely different concept (AI-judged assertion outcome vs. the runtime's own click-validity check).Verification
Before touching
wireValidation, refreshed only the vendored schema and reransrc/run/greetbot.e2e.test.ts: it failed (freshnessrequired by schema,verdictstill on the wire) — confirming the stale vendored schema, not the code, is why CI hadn't already caught this. After the full rename, the same test (and the full suite) passes.pnpm vitest run— 20 files, 187 tests passedpnpm tsc --noEmit -p tsconfig.json— clean, exit 0chatwright/chatwright'sformats/run-bundle/v1/schema.jsonsrc/for remainingverdict/Verdict— onlyAssertionVerdictand generic English "verdict" in doc comments about assertion verification remain (out of scope, correctly named)Note for Studio
Studio vendors this package by pinned commit SHA (
scripts/vendor-runtime.mjs); this PR does not bump it. A SHA bump is needed there once this merges, so Studio-emitted bundles also writefreshness.🤖 Generated with Claude Code
https://claude.ai/code/session_01SkkrXdtf8mU2GRo2hHsHT1