Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changeset/smart-insects-smile.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
"@workflow/builders": patch
"@workflow/next": patch
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about the other builders?

---

Add CFG extractor for extracting workflow graph data from bundles
55 changes: 53 additions & 2 deletions packages/builders/src/base-builder.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ const EMIT_SOURCEMAPS_FOR_DEBUGGING =
*/
export abstract class BaseBuilder {
protected config: WorkflowConfig;
protected lastWorkflowManifest?: WorkflowManifest;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm worried about sticky issues by caching this here. do we need to store intermittent state?

also currently, both, the stepsBundle and the workflowsBundle generates this manifest (I believe) - which makes this variable name odd if you're specifically trying tor ely on the step bundle

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

honestly the "partialStepManifest" logic and "partialWorkflowManifest" logic that's currently happening was hackily shipped - which is why we ended up with this split partial step and workflow manifest files that we store into the fs rather than doing it correctly

let's just take this opportunity to clean it up with some help from claude and do it right and clean :)


constructor(config: WorkflowConfig) {
this.config = config;
Expand Down Expand Up @@ -253,6 +254,7 @@ export abstract class BaseBuilder {
* Steps have full Node.js runtime access and handle side effects, API calls, etc.
*
* @param externalizeNonSteps - If true, only bundles step entry points and externalizes other code
* @returns Build context (for watch mode) and the collected workflow manifest
*/
protected async createStepsBundle({
inputFiles,
Expand All @@ -268,7 +270,10 @@ export abstract class BaseBuilder {
outfile: string;
format?: 'cjs' | 'esm';
externalizeNonSteps?: boolean;
}): Promise<esbuild.BuildContext | undefined> {
}): Promise<{
context: esbuild.BuildContext | undefined;
manifest: WorkflowManifest;
}> {
// These need to handle watching for dev to scan for
// new entries and changes to existing ones
const { discoveredSteps: stepFiles } = await this.discoverEntries(
Expand Down Expand Up @@ -389,10 +394,14 @@ export abstract class BaseBuilder {
// Create .gitignore in .swc directory
await this.createSwcGitignore();

// Store the manifest for later use (e.g., graph generation in watch mode)
this.lastWorkflowManifest = workflowManifest;

if (this.config.watch) {
return esbuildCtx;
return { context: esbuildCtx, manifest: workflowManifest };
}
await esbuildCtx.dispose();
return { context: undefined, manifest: workflowManifest };
}

/**
Expand Down Expand Up @@ -838,4 +847,46 @@ export const OPTIONS = handler;`;
// We're intentionally silently ignoring this error - creating .gitignore isn't critical
}
}

/**
* Creates a workflows manifest JSON file by extracting from the bundled workflow file.
* The manifest contains React Flow-compatible graph data for visualizing workflows.
*/
protected async createWorkflowsManifest({
workflowBundlePath,
outfile,
}: {
workflowBundlePath: string;
outfile: string;
}): Promise<void> {
const buildStart = Date.now();
console.log('Creating workflows manifest...');

try {
// Import the graph extractor
const { extractGraphFromBundle } = await import(
'./workflows-extractor.js'
);

// Extract graph from the bundled workflow file
const workflowsManifest =
await extractGraphFromBundle(workflowBundlePath);

// Write the workflows manifest
await this.ensureDirectory(outfile);
await writeFile(outfile, JSON.stringify(workflowsManifest, null, 2));

console.log(
`Created workflows manifest with ${
Object.keys(workflowsManifest.workflows).length
} workflow(s)`,
`${Date.now() - buildStart}ms`
);
} catch (error) {
console.warn(
'Failed to extract workflows from bundle:',
error instanceof Error ? error.message : String(error)
);
}
}
}
24 changes: 22 additions & 2 deletions packages/builders/src/standalone.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@ export class StandaloneBuilder extends BaseBuilder {
await this.buildWorkflowsBundle(options);
await this.buildWebhookFunction();

// Build workflows manifest from workflow bundle (post-bundle extraction)
const workflowBundlePath = this.resolvePath('.swc/workflows.js');
await this.buildWorkflowsManifest({ workflowBundlePath });

await this.createClientLibrary();
}

Expand All @@ -25,18 +29,20 @@ export class StandaloneBuilder extends BaseBuilder {
inputFiles: string[];
tsBaseUrl?: string;
tsPaths?: Record<string, string[]>;
}): Promise<void> {
}) {
console.log('Creating steps bundle at', this.config.stepsBundlePath);

const stepsBundlePath = this.resolvePath(this.config.stepsBundlePath);
await this.ensureDirectory(stepsBundlePath);

await this.createStepsBundle({
const { manifest } = await this.createStepsBundle({
outfile: stepsBundlePath,
inputFiles,
tsBaseUrl,
tsPaths,
});

return manifest;
}

private async buildWorkflowsBundle({
Expand Down Expand Up @@ -76,4 +82,18 @@ export class StandaloneBuilder extends BaseBuilder {
outfile: webhookBundlePath,
});
}

private async buildWorkflowsManifest({
workflowBundlePath,
}: {
workflowBundlePath: string;
}): Promise<void> {
const workflowsManifestPath = this.resolvePath('.swc/workflows.json');
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was considering incorporating the extraction into the steps and workflows manifest that gets generated. I think it's better to keep the generated graphs manifest separate for a couple of reasons:

  • React flow is typically the UI component used for visualizing graphs and it's a nice DX to provide a .json that has the exact schema that react flow can understand natively.
  • The goal here is primarily visualizion in the o11y dashboard and potentially showing the runs(in realtime) progressing through the graph, keeping this file separate means
  1. We can render the graph even when no workflows are run.
  2. We can neatly apply runs on top of the graph even if the runs schema changes in the future.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually don't think we should export a react flow json natively. Or atleast, we shouldn't export a manifest in the core builder that includes things like position: { x, y } etc. since that's beyond the scope of a workflow builder

it can be a simplified intermediate state that actually just represents the graph, from which the react flow graph can be rendered

I strongly think we should just have one manifest here. This split will cause more confusion and simply duplicate things we need to maintain. Let's simplify

await this.ensureDirectory(workflowsManifestPath);

await this.createWorkflowsManifest({
workflowBundlePath,
outfile: workflowsManifestPath,
});
}
}
Loading
Loading