-
Notifications
You must be signed in to change notification settings - Fork 125
Workflows CFG Extractor #455
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| --- | ||
| "@workflow/builders": patch | ||
| "@workflow/next": patch | ||
| --- | ||
|
|
||
| Add CFG extractor for extracting workflow graph data from bundles | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -26,6 +26,7 @@ const EMIT_SOURCEMAPS_FOR_DEBUGGING = | |
| */ | ||
| export abstract class BaseBuilder { | ||
| protected config: WorkflowConfig; | ||
| protected lastWorkflowManifest?: WorkflowManifest; | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm worried about sticky issues by caching this here. do we need to store intermittent state? also currently, both, the stepsBundle and the workflowsBundle generates this manifest (I believe) - which makes this variable name odd if you're specifically trying tor ely on the step bundle
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. honestly the "partialStepManifest" logic and "partialWorkflowManifest" logic that's currently happening was hackily shipped - which is why we ended up with this split partial step and workflow manifest files that we store into the fs rather than doing it correctly let's just take this opportunity to clean it up with some help from claude and do it right and clean :) |
||
|
|
||
| constructor(config: WorkflowConfig) { | ||
| this.config = config; | ||
|
|
@@ -253,6 +254,7 @@ export abstract class BaseBuilder { | |
| * Steps have full Node.js runtime access and handle side effects, API calls, etc. | ||
| * | ||
| * @param externalizeNonSteps - If true, only bundles step entry points and externalizes other code | ||
| * @returns Build context (for watch mode) and the collected workflow manifest | ||
| */ | ||
| protected async createStepsBundle({ | ||
| inputFiles, | ||
|
|
@@ -268,7 +270,10 @@ export abstract class BaseBuilder { | |
| outfile: string; | ||
| format?: 'cjs' | 'esm'; | ||
| externalizeNonSteps?: boolean; | ||
| }): Promise<esbuild.BuildContext | undefined> { | ||
| }): Promise<{ | ||
| context: esbuild.BuildContext | undefined; | ||
| manifest: WorkflowManifest; | ||
| }> { | ||
| // These need to handle watching for dev to scan for | ||
| // new entries and changes to existing ones | ||
| const { discoveredSteps: stepFiles } = await this.discoverEntries( | ||
|
|
@@ -389,10 +394,14 @@ export abstract class BaseBuilder { | |
| // Create .gitignore in .swc directory | ||
| await this.createSwcGitignore(); | ||
|
|
||
| // Store the manifest for later use (e.g., graph generation in watch mode) | ||
| this.lastWorkflowManifest = workflowManifest; | ||
|
|
||
| if (this.config.watch) { | ||
| return esbuildCtx; | ||
| return { context: esbuildCtx, manifest: workflowManifest }; | ||
| } | ||
| await esbuildCtx.dispose(); | ||
| return { context: undefined, manifest: workflowManifest }; | ||
| } | ||
|
|
||
| /** | ||
|
|
@@ -838,4 +847,46 @@ export const OPTIONS = handler;`; | |
| // We're intentionally silently ignoring this error - creating .gitignore isn't critical | ||
| } | ||
| } | ||
|
|
||
| /** | ||
| * Creates a workflows manifest JSON file by extracting from the bundled workflow file. | ||
| * The manifest contains React Flow-compatible graph data for visualizing workflows. | ||
| */ | ||
| protected async createWorkflowsManifest({ | ||
| workflowBundlePath, | ||
| outfile, | ||
| }: { | ||
| workflowBundlePath: string; | ||
| outfile: string; | ||
| }): Promise<void> { | ||
| const buildStart = Date.now(); | ||
| console.log('Creating workflows manifest...'); | ||
|
|
||
| try { | ||
| // Import the graph extractor | ||
| const { extractGraphFromBundle } = await import( | ||
| './workflows-extractor.js' | ||
| ); | ||
|
|
||
| // Extract graph from the bundled workflow file | ||
| const workflowsManifest = | ||
| await extractGraphFromBundle(workflowBundlePath); | ||
|
|
||
| // Write the workflows manifest | ||
| await this.ensureDirectory(outfile); | ||
| await writeFile(outfile, JSON.stringify(workflowsManifest, null, 2)); | ||
|
|
||
| console.log( | ||
| `Created workflows manifest with ${ | ||
| Object.keys(workflowsManifest.workflows).length | ||
| } workflow(s)`, | ||
| `${Date.now() - buildStart}ms` | ||
| ); | ||
| } catch (error) { | ||
| console.warn( | ||
| 'Failed to extract workflows from bundle:', | ||
| error instanceof Error ? error.message : String(error) | ||
| ); | ||
| } | ||
| } | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -14,6 +14,10 @@ export class StandaloneBuilder extends BaseBuilder { | |
| await this.buildWorkflowsBundle(options); | ||
| await this.buildWebhookFunction(); | ||
|
|
||
| // Build workflows manifest from workflow bundle (post-bundle extraction) | ||
| const workflowBundlePath = this.resolvePath('.swc/workflows.js'); | ||
| await this.buildWorkflowsManifest({ workflowBundlePath }); | ||
|
|
||
| await this.createClientLibrary(); | ||
| } | ||
|
|
||
|
|
@@ -25,18 +29,20 @@ export class StandaloneBuilder extends BaseBuilder { | |
| inputFiles: string[]; | ||
| tsBaseUrl?: string; | ||
| tsPaths?: Record<string, string[]>; | ||
| }): Promise<void> { | ||
| }) { | ||
| console.log('Creating steps bundle at', this.config.stepsBundlePath); | ||
|
|
||
| const stepsBundlePath = this.resolvePath(this.config.stepsBundlePath); | ||
| await this.ensureDirectory(stepsBundlePath); | ||
|
|
||
| await this.createStepsBundle({ | ||
| const { manifest } = await this.createStepsBundle({ | ||
| outfile: stepsBundlePath, | ||
| inputFiles, | ||
| tsBaseUrl, | ||
| tsPaths, | ||
| }); | ||
|
|
||
| return manifest; | ||
| } | ||
|
|
||
| private async buildWorkflowsBundle({ | ||
|
|
@@ -76,4 +82,18 @@ export class StandaloneBuilder extends BaseBuilder { | |
| outfile: webhookBundlePath, | ||
| }); | ||
| } | ||
|
|
||
| private async buildWorkflowsManifest({ | ||
| workflowBundlePath, | ||
| }: { | ||
| workflowBundlePath: string; | ||
| }): Promise<void> { | ||
| const workflowsManifestPath = this.resolvePath('.swc/workflows.json'); | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I was considering incorporating the extraction into the steps and workflows manifest that gets generated. I think it's better to keep the generated graphs manifest separate for a couple of reasons:
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I actually don't think we should export a react flow json natively. Or atleast, we shouldn't export a manifest in the core builder that includes things like it can be a simplified intermediate state that actually just represents the graph, from which the react flow graph can be rendered I strongly think we should just have one manifest here. This split will cause more confusion and simply duplicate things we need to maintain. Let's simplify |
||
| await this.ensureDirectory(workflowsManifestPath); | ||
|
|
||
| await this.createWorkflowsManifest({ | ||
| workflowBundlePath, | ||
| outfile: workflowsManifestPath, | ||
| }); | ||
| } | ||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what about the other builders?