feat: multi-trace induction — infer a parameterized program (params/loops/branches) from multiple demos, reject-if-underdetermined#81
Merged
Conversation
…ion paths (the state machine) Evolve the compiled artifact from a linear action list into a parameterized STATE MACHINE (RFC docs/design/WORKFLOW_PROGRAM_IR.md §2), closing the review's "a workflow is not a list of actions" gap. Phase 1 (typed params, guards, wait_until) added the pieces; Phase 2 adds the control flow a trajectory cannot carry: LOOPS over a worklist, guarded BRANCHES, reusable SUBFLOWS, and EXCEPTION paths — the program the PBD literature (Rousillon, WebRobot, Skill-DisCo, PROLEX) says a demonstration compiler must express. IR (openadapt_flow/ir.py), additive and backward-compatible: - State (action | branch | loop | subflow_call | terminal) + Transition (guarded edge) form a ProgramGraph; an action state's payload IS a Phase-1 Step (the unchanged hardened leaf), a transition's guard IS a Phase-1 Predicate. - Relation (worklist) + LoopSpec (bounded per-row body subflow); Workflow gains optional program / subflows / data_sources. When program is None the linear steps list runs exactly as today. - lift_to_program: mechanical degenerate lift (RFC §2.6) — a linear bundle is the single-path graph. Interpreter (runtime/replayer.py): a deterministic graph interpreter ($0, zero model calls) that REUSES the linear per-action pipeline unchanged — every action state runs through _run_step, so identity / effect / risk / heal gates fire identically inside loop bodies and branches. Adds guarded transition selection (first match wins, no-match HALTs fail-safe), bounded worklist loops, subflow dispatch, and on_exception routing (graph try/except); unhandled failures and halt/escalate terminals stop the run. Bounded against non-terminating graphs (step budget + nesting depth). Linear path is byte-for- byte unchanged (program=None branch). Tests (tests/test_program_ir_phase2.py, 18): loop runs body 3x / 0x / run-time worklist / bound enforced; branch takes each arm (param + screen predicate) and dead-ends HALT; subflow reused as loop body AND direct call; on_exception catches a failed action and continues; identity- and effect-gates fire inside a loop body; the lifted linear graph replays byte-identically to the linear replayer; program round-trips through save/load. Full non-e2e suite green in isolation (859 passed; the concurrent-agent FileNotFoundError errors are environmental). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ
…oops/branches) from multiple demos, reject-if-underdetermined Implements RFC docs/design/WORKFLOW_PROGRAM_IR.md §3 steps [4]+[5]: the induction loop the PBD lineage (Rousillon, WebRobot, Skill-DisCo, PROLEX) says a demonstration compiler must have. "One demonstration is evidence, not specification." openadapt_flow/compiler/induction.py: - induce_program(traces) aligns multiple demos structurally and infers a Phase-2 ProgramGraph: PARAMS (values that VARY across traces at an aligned position; constant => literal), LOOPS (a repeated body whose count DIFFERS => LoopSpec over an inferred Relation), BRANCHES (a divergent step under a detectable condition => guarded branch, guard proposed/flagged), and OPTIONAL steps (present in some, absent in others, no condition => guarded skip). All deterministic, ZERO model calls. - validate_held_out / reproduction_score: leave-one-out held-out validation (infer from N-1, check reproduction of the held trace). - Reject-rather-than-guess: contradictory / underdetermined traces are QUARANTINED (no program emitted, certified=False) and routed to the disambiguation flow (#74), mirroring the identity gate's posture. - The optional compile-time Proposer (the #78 StepAnnotator fits behind it) only PROPOSES interpretations — flagged, never silently trusted, never flips an underdetermined point to certified. Touch-points kept minimal: reuses the Phase-2 IR + Phase-1 ParamSpec/Guard/ Predicate verbatim (no new IR fields), reuses disambiguation's question model, and the emitted program replays through the EXISTING interpreter unchanged (compile.py untouched; compiler/__init__ re-exports the new API). Tests (tests/test_induction.py, 17 tests): a synthetic MockMed corpus of trace variants covers (a) param, (b) loop, (c) branch/optional, (d) contradiction=> reject; held-out scores a good induction high and an over-specialized one low; underdetermined is flagged not guessed; the induced program round-trips through the real Phase-2 interpreter (faked backend/vision, zero model calls). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ
…ction # Conflicts: # openadapt_flow/compiler/__init__.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Multi-trace induction: infer a parameterized workflow PROGRAM (the Phase-2
ProgramGraph) from MULTIPLE demonstrations of the same task — the induction loop the whole PBD lineage (Rousillon, WebRobot, Skill-DisCo, PROLEX) says a demonstration compiler must have. Implements RFCdocs/design/WORKFLOW_PROGRAM_IR.md§3 steps [4] multi-trace induction + [5] held-out validation / quarantine. "One demonstration is evidence, not specification."The induction loop (
openadapt_flow/compiler/induction.py)induce_program(traces: list[Workflow | recording-dir]) -> InductionResult:ParamSpec; CONSTANT → a baked literal.LoopSpecover an inferredRelation, body = the repeated subflow.branchstate with guarded transitions (guard proposed, flagged for confirmation).ProgramGraph(the Phase-2 IR) — replayed through the EXISTING interpreter unchanged.Reject-rather-than-guess (§3 [5])
When traces CONTRADICT or intent is underdetermined (a divergent branch with no detectable condition, unrelated traces), induction does not fabricate: it marks the point
underdeterminedwith the specific ambiguity, routes it to the disambiguation flow (#74), leavescertified=False, and does not emit a program. Mirrors the identity gate's refuse-rather-than-guess posture.Deterministic core vs proposes-and-flags
Structural inference is the core. An optional compile-time
Proposer(the #78StepAnnotatorfits behind this interface) may PROPOSE an interpretation, but every proposal is flagged, never silently trusted and can never flip an underdetermined point to certified. Tests use a deterministic fake (zero model calls).Held-out validation
validate_held_out(leave-one-out) +reproduction_score: infer from N-1 traces, check the induced program reproduces the held-out trace. A good (param) induction scores high; an over-specialized one (value frozen as a literal because the demos didn't vary it) scores low on a held trace with a different value.Synthetic corpus + tests (
tests/test_induction.py, 17 tests)Trace-variant generators of a MockMed task: (a) varying value → param, (b) worklist 2 vs 3 → loop, (c) optional Survey dialog present vs absent → branch, (d) approve-vs-reject contradiction → REJECT. Held-out good-high/over-specialized-low; underdetermined flagged not guessed; the induced program round-trips through the real Phase-2 interpreter (faked backend/vision from
test_replayer, zero model calls). Related suites green (induction + phase1/phase2 IR + disambiguation + compiler = 106 tests).Touch-points (minimal, per the stacking constraint)
compile.pyUNCHANGED (it is the single-trace bootstrap, §3 [1]);compiler/__init__re-exports the new API.ParamSpec/Guard/Predicateverbatim — no new IR fields. Reusesdisambiguation's question model.Stacking
Stacks on #79 (
feat/workflow-program-ir-phase2) for the Phase-2 state-machine IR. Base isfeat/workflow-program-ir-phase2; retarget tomainafter #79 merges.🤖 Generated with Claude Code
https://claude.ai/code/session_01CKrVJJy5jWVCkXAqgUqtqZ