epic: unify composition primitives and graph execution model

[nextlevelshit]

Context

Wave has two overlapping flow control systems that evolved independently:

System 1: Composition Primitives (pre-epic #589)

• iterate — fan-out over items (sequential or parallel)
• branch — route based on context variable
• loop — repeat steps until condition met
• aggregate — merge results from fan-out
• sub_pipeline (via pipeline: field) — invoke child pipeline

Used by: audit-quality-loop, ops-epic-runner, ops-implement-epic, ops-parallel-audit, ops-release-harden
Visible in: WebUI /compose page

System 2: Graph Execution Model (epic #589)

• type: conditional — route based on step outcome via edges
• type: command — shell script execution
• max_visits — loop safety limit
• edges with backward targets — creates cycles
• type: pipeline — sub-pipeline invocation

Used by: impl-hotfix, wave-bugfix, ops-debug, test-gen, wave-test-hardening, wave-validate
Visible in: WebUI DAG with step type shapes

The Overlap

Capability	Composition	Graph
Conditional routing	branch: with cases	type: conditional + edges
Looping	loop: with until	edges pointing backward + max_visits
Sub-pipeline	pipeline: field	type: pipeline
Fan-out/parallel	iterate: with mode: parallel	(no equivalent)
Aggregation	aggregate:	(no equivalent)
Shell execution	(no equivalent)	type: command

Why This Matters

• Pipeline authors must learn two models
• Validator needs to handle both
• WebUI renders them differently (compose page vs DAG step types)
• New features must be implemented in both systems or one gets left behind
• The executor has two code paths (executeStep vs executeGraphPipeline)

Research Questions

1. Can graph features subsume composition primitives? Graph edges + conditional routing could replace branch. max_visits + backward edges could replace loop. But what about iterate (parallel fan-out) and aggregate?

2. Can composition primitives subsume graph features? loop could replace max_visits + edges. branch could replace type: conditional. But composition primitives are higher-level — they don't support arbitrary DAG shapes.

3. Should we keep both and formalize the boundary? Composition = orchestration layer (pipeline-of-pipelines). Graph = step layer (within a single pipeline). Clear separation, no overlap.

4. What do users lose in each unification direction?

5. Migration path: 6 composition pipelines + 6 graph pipelines need migration or the old system needs maintained indefinitely.

Scope

Phase 1: Research

• Map every composition primitive to its graph equivalent (or document why no equivalent exists)
• Interview pipeline authors (check existing pipeline patterns) — which model do they reach for?
• Benchmark: does iterate parallel fan-out have performance characteristics that graph mode can't match?
• Survey other tools: how do Fabro, GitHub Actions, Argo Workflows handle this?

Phase 2: ADR

• Write ADR documenting the chosen direction (unify to graph, unify to composition, or formalize boundary)
• Define migration plan for affected pipelines

Phase 3: Implementation

• Execute the chosen direction
• Migrate affected pipelines
• Update WebUI (compose page and DAG view converge or differentiate)
• Update documentation

Validation Criteria

• One clear model for pipeline authors (or two with a documented boundary)
• All existing pipelines continue to work
• WebUI reflects the unified/clarified model
• Documentation explains when to use what

[nextlevelshit]

Research Findings: Composition Primitives vs Graph Execution

1. Can Graph Mode Express Each Composition Primitive?

iterate with mode: parallel — NO, graph mode cannot fan out

Graph mode is fundamentally single-threaded. GraphWalker.Walk() follows one edge at a time — currentStepID is a single string, not a set. There is no mechanism to spawn N concurrent step executions from a dynamic list of items.

To replicate iterate in graph mode you would need:

• A "splitter" step that emits N edge targets — but edge targets must be statically declared in YAML
• Or a meta-step that dynamically generates steps — which is what iterate already is

Verdict: Graph mode would need a new fan_out / parallel_for primitive. This is not a trivial extension — it requires concurrent execution within the graph walker, result collection, and error aggregation. The composition iterate already does this via errgroup.

aggregate with merge_arrays — NO, graph mode has no data collection primitive

aggregate operates on outputs from a fan-out step (reading {{ run-audits.output }}), merging JSON arrays, and writing the result to disk. Graph mode has no mechanism for:

• Collecting outputs from multiple prior step executions
• Merging structured data
• The template context (StepContext) only carries Outcome (string) and Context (map[string]string) — no structured data

Verdict: Graph mode would need a new reduce / collect step type with structured data handling.

loop with until — YES, graph mode handles this well

Graph mode's edges pointing backward + max_visits is a clean loop implementation. Compare:

# Composition loop
loop:
  max_iterations: 3
  until: "{{supervise.output.verdict}}"
  steps: [improve, recheck]

# Graph equivalent (already used in impl-hotfix, wave-bugfix)
- id: run-tests
  type: command
  script: "{{ project.contract_test_command }}"
- id: gate
  type: conditional
  edges:
    - target: verify
      condition: "outcome=success"
    - target: fix      # backward edge = loop

Graph mode is actually more expressive here — it can route to different steps on different conditions, whereas loop.until is a single boolean check. Graph mode also has the circuit breaker (checkCircuitBreaker) that trips after 3 identical errors, which composition loop lacks.

Verdict: Graph mode subsumes composition loop and adds safety features.

branch with cases — PARTIALLY, different routing models

Composition branch routes to different sub-pipelines based on a template variable:

branch:
  on: "{{scan.output.risk_level}}"
  cases:
    critical: impl-hotfix
    high: impl-hotfix
    medium: impl-improve
    low: skip

Graph type: conditional routes to different steps within the same pipeline based on outcome=success/failure or context.KEY=VALUE:

type: conditional
edges:
  - target: verify
    condition: "outcome=success"
  - target: fix

Key differences:

• Composition branch dispatches to named pipelines (cross-pipeline routing)
• Graph conditional dispatches to steps within the same pipeline
• Composition branch evaluates arbitrary template expressions; graph conditions are limited to outcome and context.KEY equality checks
• Composition branch supports a skip sentinel; graph mode requires an explicit no-op step

Verdict: Neither fully subsumes the other. Graph conditions are more limited but embedded in the execution flow. Composition branch is more flexible but only operates at the pipeline-of-pipelines level.

---

2. What Do Users LOSE If We Remove Composition Primitives?

1. Parallel fan-out over dynamic lists (iterate mode: parallel) — this is the killer feature. Graph mode cannot do this without a new primitive. 3 out of 6 composition pipelines use iterate.

2. Data aggregation (aggregate merge_arrays) — collecting and merging structured outputs. Graph mode's StepContext is flat key-value pairs, not suitable for structured data merging.

3. Cross-pipeline routing (branch dispatching to named pipelines) — graph mode only routes within a single pipeline. The composition branch in ops-release-harden dispatches to impl-hotfix or impl-improve based on risk level.

4. Declarative loop body (loop.steps) — composition loops declare sub-steps inline. Graph loops require manually wiring edges, which is more verbose but also more flexible.

5. Pipeline-of-pipelines ergonomics — composition pipelines read like orchestration scripts: "iterate over issues, implement each, then aggregate." Graph pipelines read like state machines.

3. What Do Users LOSE If We Remove Graph Features?

1. Cycle-aware retry loops — the pattern of run-tests -> gate (conditional) -> fix (backward edge) is used in 5 production pipelines. Composition loop can approximate this but requires nesting the entire retry body as loop.steps, which is less natural.

2. Circuit breaker — graph walker trips after 3 identical consecutive errors. Composition loop has no equivalent safety mechanism.

3. Shell-only steps (type: command) — lightweight steps that don't need an adapter/persona. However, the DAG executor ALSO supports command steps (lines 1355-1366 in executor.go), so this is not exclusive to graph mode.

4. Visit counting and resume — GraphWalker.VisitCounts() enables precise state tracking for pipeline resumption. Composition loop tracks iteration count but not per-step visit counts.

5. Arbitrary conditional routing — graph edges can route to any step based on outcome or context, not just "loop back" or "branch to pipeline."

4. Is There a Clean Boundary?

Yes, and the codebase already hints at it:

Layer	System	Operates On	Purpose
Orchestration	Composition	Pipelines	Coordinate multiple pipelines: fan-out, aggregate, branch across pipelines
Execution	Graph	Steps within one pipeline	Control step flow: conditional routing, retry loops, command steps

Evidence:

• Composition steps reference pipelines by name (pipeline: impl-issue), graph steps reference other steps by ID (target: fix)
• CompositionExecutor.runSubPipeline() delegates to SequenceExecutor.Execute() which delegates to DefaultPipelineExecutor.Execute() — composition is a layer above
• Graph mode is activated by IsGraphPipeline() which checks for edges/conditional types — it's a property of a single pipeline's step structure
• The category: composition metadata on composition pipelines already signals they operate at a different level

The overlap exists specifically in these two areas:

1. Looping: loop.until (composition) vs backward edges + max_visits (graph) — both loop within a pipeline
2. Branching: branch.cases (composition, dispatches to pipelines) vs type: conditional (graph, dispatches to steps) — both make routing decisions

5. How Other Tools Handle This

GitHub Actions: Reusable Workflows vs Job Conditions

GitHub Actions has a clean two-level model:

• Job level: jobs.<id>.if conditions, needs dependencies, strategy.matrix for fan-out — this is their "graph" layer
• Reusable Workflows: uses: ./.github/workflows/foo.yml — this is their "composition" layer, one workflow calling another
• Key insight: They do NOT unify these. Matrix fan-out lives at the job level. Reusable workflows are a separate invocation mechanism. Conditions (if:) work at both levels but with different scopes.

Argo Workflows: DAG vs Steps Templates

Argo Workflows explicitly maintains two execution models:

• dag template: Steps with dependencies, parallel by default, no cycles. Similar to Wave's DAG mode.
• steps template: Sequential/parallel groups, supports when conditions. Similar to Wave's composition.
• Key insight: Argo keeps both and documents when to use which. DAG for complex dependency graphs, Steps for simple sequential/parallel flows. They also have withItems (iterate) and withParam (dynamic fan-out) available in both modes.

Both GitHub Actions and Argo validate Wave's current dual-model approach. Neither tool chose to unify into a single model.

---

Recommendation

Formalize the boundary. Do not unify.

Rationale:

1. Graph mode cannot express parallel fan-out or aggregation without becoming a different system. Adding these to graph mode would recreate composition primitives under different names.
2. Composition mode cannot express arbitrary step-level routing or cycle-aware retry without becoming a graph system.
3. The two models serve different pipeline author personas: orchestrators (who think in pipelines and data flow) vs implementers (who think in steps and control flow).
4. Industry precedent (GitHub Actions, Argo Workflows) supports maintaining two models with clear boundaries.

Specific actions:

1. Document the boundary explicitly — composition = pipeline-of-pipelines orchestration, graph = intra-pipeline control flow
2. Remove the two areas of overlap:
   • Deprecate loop in composition (use graph backward edges + max_visits instead when you need loops within a single pipeline)
   • Ensure type: command works identically in both modes (it already does — lines 920-921 and 1355-1366 in executor.go both call executeCommandStep)
3. Keep branch in composition since it dispatches across pipelines (graph conditional dispatches within one pipeline) — these are genuinely different operations despite surface similarity
4. Add fan-out to graph mode as a future enhancement if needed, but do not remove composition iterate/aggregate — they are the unique value of the composition layer
5. Update WebUI to render composition pipelines as "orchestration flows" and graph pipelines as "step graphs" — making the conceptual difference visible

[nextlevelshit]

ADR-007 written: formalize boundary between composition primitives (pipeline-level orchestration) and graph execution (step-level routing). Both systems maintained, no migration needed. See docs/adrs/007-composition-graph-boundary.md

Commit: 251ca0c