fix(next): skip eager workflow discovery in deferred mode#1585
Closed
fix(next): skip eager workflow discovery in deferred mode#1585
Conversation
🦋 Changeset detectedLatest commit: 0c9ecfb The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
📊 Benchmark Results
workflow with no steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) workflow with 1 step💻 Local Development
▲ Production (Vercel)No data available workflow with 10 sequential steps💻 Local Development
▲ Production (Vercel)No data available workflow with 25 sequential steps💻 Local Development
▲ Production (Vercel)No data available workflow with 50 sequential steps💻 Local Development
▲ Production (Vercel)No data available Promise.all with 10 concurrent steps💻 Local Development
▲ Production (Vercel)No data available Promise.all with 25 concurrent steps💻 Local Development
▲ Production (Vercel)No data available Promise.all with 50 concurrent steps💻 Local Development
▲ Production (Vercel)No data available Promise.race with 10 concurrent steps💻 Local Development
▲ Production (Vercel)No data available Promise.race with 25 concurrent steps💻 Local Development
▲ Production (Vercel)No data available Promise.race with 50 concurrent steps💻 Local Development
▲ Production (Vercel)No data available workflow with 10 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)No data available workflow with 25 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)No data available workflow with 50 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)No data available workflow with 10 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)No data available workflow with 25 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)No data available workflow with 50 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)No data available Stream Benchmarks (includes TTFB metrics)workflow with stream💻 Local Development
▲ Production (Vercel)No data available stream pipeline with 5 transform steps (1MB)💻 Local Development
▲ Production (Vercel)No data available 10 parallel streams (1MB each)💻 Local Development
▲ Production (Vercel)No data available fan-out fan-in 10 streams (1MB each)💻 Local Development
▲ Production (Vercel)No data available SummaryFastest Framework by WorldWinner determined by most benchmark wins
Fastest World by FrameworkWinner determined by most benchmark wins
Column Definitions
Worlds:
❌ Some benchmark jobs failed:
Check the workflow run for details. |
Contributor
🧪 E2E Test Results❌ Some tests failed Summary
❌ Failed Tests▲ Vercel Production (63 failed)nextjs-turbopack (63 failed):
📦 Local Production (97 failed)nextjs-turbopack-canary (40 failed):
nextjs-turbopack-stable (57 failed):
🐘 Local Postgres (97 failed)nextjs-turbopack-canary (40 failed):
nextjs-turbopack-stable (57 failed):
🪟 Windows (57 failed)nextjs-turbopack (57 failed):
🌍 Community Worlds (62 failed)mongodb-dev (1 failed):
redis-dev (1 failed):
turso-dev (1 failed):
turso (59 failed):
Details by Category❌ ▲ Vercel Production
✅ 💻 Local Development
❌ 📦 Local Production
❌ 🐘 Local Postgres
❌ 🪟 Windows
❌ 🌍 Community Worlds
✅ 📋 Other
❌ Some E2E test jobs failed:
Check the workflow run for details. |
Member
VaguelySerious
left a comment
VaguelySerious
left a comment
There was a problem hiding this comment.
AI review: blocking issues found
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
…re explicit In deferred build mode, workflow discovery should only happen through: 1. Cache from previous builds 2. Loader socket notifications when Next.js loads files When users specify explicit workflow directories (workflows.dirs option), the dirsAreEntrypoints flag is false, indicating these are actual workflow directories, not entrypoints to trace from. Previously, loadDiscoveredEntriesFromInputGraph() was always called during initialization, performing eager discovery that defeats the purpose of both: - Deferred mode (which should discover on-demand) - The workflows.dirs optimization (which avoids scanning the entire app) This fix conditionally skips eager discovery when dirsAreEntrypoints is false, allowing deferred mode to work as intended with explicit workflow directories.
The dirsAreEntrypoints property was being used but wasn't defined in the NextConfig interface. This adds: - dirsAreEntrypoints property to NextConfig in types.ts - Type import and declaration in builder-deferred.ts - Type import and declaration in builder-eager.ts (for consistency) This fixes the TypeScript compilation error in CI.
In deferred mode, workflow discovery should ONLY happen through: 1. Cache restoration from previous builds 2. Loader socket notifications when Next.js loads files The previous fix was overly complex and tied to the dirs config. The real issue is simpler: deferred mode should never do eager discovery at all, regardless of configuration. Changes: - Removed loadDiscoveredEntriesFromInputGraph() call from initializeDiscoveryState() - Removed dirsAreEntrypoints type additions (not needed) - Added test to verify 'Discovering workflow directives' log is not printed - Simplified comments to clarify deferred vs eager discovery behavior
The @workflow/ai package was exporting compiled JS which strips 'use step' directives. This caused internal step functions like closeStream to not be registered in deferred mode. Solution: Add 'workflow' export condition that points to the TypeScript source, similar to how @workflow/core and workflow packages work. This ensures the workflow bundler processes the source with directives intact.
The src folder is not published with the package, so pointing the workflow export to src/agent/durable-agent.ts causes resolution errors. The compiled JS preserves 'use step' directives, so we can safely point the workflow export to the dist file. This matches the pattern where workflow and default exports are the same (unlike @workflow/core which has separate workflow-specific builds).
The extractBundleSourceFiles method was excluding ALL node_modules files from tracked dependency discovery. This prevented step functions in @workflow/ai from being discovered after bundling. Solution: Allow @workflow/* scoped packages to be tracked as dependencies, while still excluding other node_modules. This ensures internal step functions in workflow SDK packages are properly discovered and registered.
Next.js doesn't process node_modules files with custom loaders by default. Adding @workflow/ai to transpilePackages ensures our loader processes it, allowing step functions like closeStream to be discovered via socket notifications.
In production builds, socket notifications don't trigger rebuilds because scheduleDeferredRebuild() is a no-op when watch is false. Without initial discovery, there are no files to bundle, so the system can't bootstrap. Solution: Perform discovery in two cases: 1. Production builds (always, because socket rebuilds don't work) 2. First dev build when cache is empty (to bootstrap) In dev mode with cache, skip discovery and rely on socket notifications. This provides the optimization for the common case (dev with cache) while ensuring production builds and first builds work correctly.
1. Fix test to use mocking instead of dynamic import (avoids VM error) 2. Include discoveredSerdeFiles in cache check 3. Apply @workflow package allowlist to shouldSkipTransitiveStepFile 4. Add comment about @workflow/ai being optional dependency 5. Make changeset more concise
The previous test mocked the entire module and re-implemented initializeDiscoveryState in the mock, so it tested mock behavior rather than production code. New approach: Spy on BaseBuilder.discoverEntries to verify when discovery is called, while testing the real initializeDiscoveryState implementation. This ensures tests will fail if the production code changes incorrectly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Workflow discovery was happening on every build in deferred mode, even in dev mode with an existing cache. This defeats the optimization purpose of deferred mode.
Root Cause
The deferred builder was calling
loadDiscoveredEntriesFromInputGraph()unconditionally during initialization, performing eager discovery on every build regardless of context.Solution
Made discovery conditional based on build context:
Skip discovery when:
watch: true) AND cache existsPerform discovery when:
Additional fixes:
@workflow/*packages in tracked dependency discovery (bothextractBundleSourceFilesandshouldSkipTransitiveStepFile)@workflow/aitotranspilePackagesso loader processes it (optional dependency, Next.js handles gracefully if not installed)"workflow"export condition to@workflow/ai/agent(points to same dist file as default - forward compatibility for potential future workflow-specific builds)Changes
packages/next/src/builder-deferred.ts: Conditional discovery based on watch mode and cache, consistent @workflow package filteringpackages/next/src/index.ts: Add @workflow/ai to transpilePackages with explanatory commentpackages/ai/package.json: Added workflow export conditionpackages/next/src/builder-deferred.test.ts: Mocked tests avoiding dynamic import issuesImpact
This provides the optimization for the common dev workflow:
Related
Slack Thread