Add forEach step type for VMCP composite tool workflows#4375
Conversation
There was a problem hiding this comment.
Large PR Detected
This PR exceeds 1000 lines of changes and requires justification before it can be reviewed.
How to unblock this PR:
Add a section to your PR description with the following format:
## Large PR Justification
[Explain why this PR must be large, such as:]
- Generated code that cannot be split
- Large refactoring that must be atomic
- Multiple related changes that would break if separated
- Migration or data transformationAlternative:
Consider splitting this PR into smaller, focused changes (< 1000 lines each) for easier review and reduced risk.
See our Contributing Guidelines for more details.
This review will be automatically dismissed once you add the justification section.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #4375 +/- ##
==========================================
+ Coverage 69.42% 69.52% +0.10%
==========================================
Files 485 485
Lines 49560 49789 +229
==========================================
+ Hits 34405 34616 +211
- Misses 12489 12494 +5
- Partials 2666 2679 +13 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
5b6ac44 to
13ae52c
Compare
13ae52c to
9b60a62
Compare
9b60a62 to
ee0ab74
Compare
Workflows today require statically defined steps -- you cannot dynamically spawn steps based on a previous step's output. This makes patterns like "for each package in the image, query for vulnerabilities" impossible without hardcoding every item. Add a third step type `forEach` that iterates over a collection produced by a previous step and executes an inner tool step for each item, with configurable parallelism and error handling. Key design decisions: - forEach is a DAG-opaque macro: the DAG sees one node, internal parallelism is self-managed via errgroup + semaphore - Single inner step (tool type only) keeps it simple; multi-step iteration bodies and elicitation inner steps can come later - Nested forEach is explicitly rejected at validation time - Collection resolves via Go template expansion then JSON parse, reusing existing template infrastructure - Aggregated output has a well-known shape (iterations, count, completed, failed) so downstream templates have a predictable structure to reference Safety limits: - maxIterations: default 100, hard cap 1000 - maxParallel: default 10 (DAG default), hard cap 50 - itemVar cannot shadow the reserved "index" key - Step and workflow timeouts apply as normal Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ee0ab74 to
2cf35b8
Compare
yrobla
left a comment
There was a problem hiding this comment.
Two findings in executeForEachStep — one functional issue worth addressing before merge, one structural nit that can be a follow-up.
Write a "cancelled" sentinel to iteration results when errgroup cancels the context, preventing zero-value entries from producing misleading aggregated output. Add MaxParallel() accessor to dagExecutor to avoid coupling forEach to DAG struct internals. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Summary
forEachthat iterates over a collection produced by a previous step and executes an inner tool step for each item, with configurable parallelism and error handling.Type of change
Test plan
task test)task lint-fix)Changes
pkg/vmcp/config/config.goWorkflowStepConfig, update kubebuilder enumpkg/vmcp/config/composite_validation.goValidateForEachStepwith inner step, collection, limits validationpkg/vmcp/config/foreach_validation_test.gopkg/vmcp/composer/composer.goStepTypeForEach, forEach fields onWorkflowStep, 2 newTemplateExpandermethodspkg/vmcp/composer/template_expander.goextraCtxfor forEachpkg/vmcp/composer/workflow_engine.goexecuteForEachStepwith collection resolution, bounded parallel execution, result aggregationpkg/vmcp/composer/foreach_test.gopkg/vmcp/server/workflow_converter.gopkg/vmcp/internal/compositetools/workflow_converter.godocs/arch/10-virtual-mcp-architecture.mddocs/operator/composite-tools-quick-reference.mddocs/operator/virtualmcpcompositetooldefinition-guide.mddocs/operator/advanced-workflow-patterns.mdDoes this introduce a user-facing change?
Yes. Users can now define
forEachsteps in composite tool workflows to iterate over collections from previous steps. Example:Special notes for reviewers
Key design decisions:
forEachis a DAG-opaque macro -- the DAG sees one node; internal parallelism is self-managed via errgroup + semaphore. This avoids rearchitecting the DAG executor.expandValueWithDepthandexpandStringInternalnow accept an optionalextraCtxparameter, used by both normal and forEach expansion paths.Safety limits:
maxIterations: default 100, hard cap 1000maxParallel: default 10 (DAG default), hard cap 50itemVarcannot be"index"(reserved key)Interface change: Two methods added to
TemplateExpander(ExpandString,ExpandWithForEach).ExpandStringis a natural complement toExpand(string vs map).ExpandWithForEachcleanly separates the forEach concern without mutating sharedWorkflowContextstate. Both are backed by the same unified internal implementation.Large PR Justification
foreach_test.go,foreach_validation_test.go).Generated with Claude Code