agent_sdk+scripts: explore videos, capture-verdict outcomes, deepest-failure naming, synthesis prompt decision (sysID stack 7/8) - #102
Merged
Conversation
Interaction videos were concatenated into a single per-cycle file written only at cycle end, so a mid-cycle crash lost all footage and no transcript could be paired with its own episode. Each episode's video is now saved as soon as the episode ends, named __ep<i>__cycle<C>.mp4; log_viewer pairs each explore transcript with its own episode's video via the session-order index (legacy whole-cycle videos still render with the old shared-video label).
Episode rows were colored by the last 'Goal achieved: True|False' line, which is a per-rollout goal-atom check that stays True even when the evaluator rejects the plan (solved=False) - sessions full of rejected submissions rendered as successes. When a session used evaluate_option_plan, its capture verdicts now decide the outcome: any clean 'Captured as the current answer' means the session holds an answer, best-effort captures and NOT CAPTURED do not; sessions without capture verdicts keep the old goal-line fallback.
…ned plan The truncation log line said only how many steps survived; it now appends which option failed deepest and why, so a truncated experiment plan can be diagnosed from the run log without replaying the search.
Three 45-minute attempts already exhaust productive search per test task (later attempts rediscover the journal's dead ends), and the heavy-block env now ships turn-only test tasks (domino_test_turn_ratio 1.0) to match the arm the certificates were retuned for.
… explicit decision Trigger: run_20260725_223530 seed1 went 0/1 for four-plus cycles on the same test task its sibling run solved in one, because the synthesis agent never declared PHYSICAL_PARAMS - no sysID ran, the belief env stayed at the planning friction, and the evaluator's rules-free verification replay rejected every goal-reaching plan (335 probe rejections, 0 passes), which the agent then misread as a task rule. Cycle-0's omission was never revisited: later fresh-context sessions inherit simulator.py and the prompt framed it as the starting point. Three prompt changes: - The sysID section is retitled 'decide, don't default' and states the one non-inferable consequence of omission: undeclared parameters keep their built-in values in every base-sim rollout, including checks that re-simulate the scene without the agent's rules (e.g. the evaluator's verification replay deciding what counts as a solve). - simulator.py must open with a DECISION RECORD comment (base-sim vs rules split, feature ownership, latents, other structural commitments) so later cycles re-read the rationale instead of silently inheriting it. - The prior-cycle block says structural decisions are not binding across cycles and that goal-reaching episodes scored solved=0 point to a structural modeling error, not parameter values.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Seventh of eight PRs landing the domino-sysid branch on master in sequence (after #96-#101). Tree is identical to the corresponding slice of that branch rebased onto current master.
This chunk
Observability and prompt-decision work: per-episode explore videos, log_viewer scoring episodes by capture verdicts, refined-plan truncation reporting the deepest validation failure instead of a generic message, and the synthesis prompt turning base-sim parameter identification into an explicit yes/no decision the agent must justify.
🤖 Generated with Claude Code