Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs-web/architecture/node-flow-foundation.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ A validated custom definition becomes executable only after its immutable artifa

## Validation, policy, and secrets

Validation resolves each definition and checks configuration, port handles and schemas, policies, graph bounds, and cycles. It rejects plaintext secret-shaped fields and generated or custom source in graph JSON. The dashboard receives credential binding ids and metadata-only states; resolved values remain behind the credential broker. Credential values and secret-shaped payloads are redacted before invocation messages, attempts, diagnostics, route responses, and debugger output are persisted or rendered.
Validation resolves each definition and checks configuration, port handles and schemas, policies, graph bounds, and cycles. Migrated Graph v1 and canonical Graph v2 inputs fail closed: malformed nodes, edges, ports, credential bindings, definition references, capabilities, policies, schemas, and metadata produce deterministic field-level issues at their original paths without discarding safe siblings. Repeated validation preserves issue ordering. It rejects plaintext secret-shaped fields and generated or custom source in graph JSON. The dashboard receives credential binding ids and metadata-only states; resolved values remain behind the credential broker. Credential values and secret-shaped payloads are redacted before invocation messages, attempts, diagnostics, route responses, and debugger output are persisted or rendered.

Draft review reports requested capabilities, side effects, credential status, policy findings, and a non-executing dry run. Publication requires the current `draftRevision`, a valid graph and policy review, and every required credential binding. Runs select an immutable pinned or latest-published snapshot.

Expand Down
2 changes: 1 addition & 1 deletion docs-web/architecture/node-flows.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Authenticated dashboard routes resolve persisted project ownership from flow, ru
| `webhook_trigger` | Emits secret-authenticated webhook input. |
| `output` | Selects the result. |

These are the executable definitions. Other custom palette concepts remain non-executable until a versioned handler is registered. Graph v1 migration preserves the legacy snapshot and appends deterministic v2.
These are the executable definitions. Other custom palette concepts remain non-executable until a versioned handler is registered. Graph v1 migration preserves the legacy snapshot and appends deterministic v2. Both migrated v1 and canonical v2 graphs are validated as untrusted input: malformed nested members fail closed with stable paths at their original array indices, valid siblings are retained where safe, and repeated validation returns the same ordered issues instead of throwing.

Execution uses immutable publications rather than the mutable editor row. Runs select a pinned publication or the latest published version, then use durable queue claims, leases, bounded quotas, timeout/cancellation propagation, and numbered retry attempts. Expired external attempts with unknown outcomes require operator attention and are not silently replayed. See [Node Flow Durable Execution](./node-flow-durable-execution.md).

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ A validated custom definition becomes executable only after its immutable artifa

## Validation, policy, and secrets

Validation resolves each definition and checks configuration, port handles and schemas, policies, graph bounds, and cycles. It rejects plaintext secret-shaped fields and generated or custom source in graph JSON. The dashboard receives credential binding ids and metadata-only states; resolved values remain behind the credential broker. Credential values and secret-shaped payloads are redacted before invocation messages, attempts, diagnostics, route responses, and debugger output are persisted or rendered.
Validation resolves each definition and checks configuration, port handles and schemas, policies, graph bounds, and cycles. Migrated Graph v1 and canonical Graph v2 inputs fail closed: malformed nodes, edges, ports, credential bindings, definition references, capabilities, policies, schemas, and metadata produce deterministic field-level issues at their original paths without discarding safe siblings. Repeated validation preserves issue ordering. It rejects plaintext secret-shaped fields and generated or custom source in graph JSON. The dashboard receives credential binding ids and metadata-only states; resolved values remain behind the credential broker. Credential values and secret-shaped payloads are redacted before invocation messages, attempts, diagnostics, route responses, and debugger output are persisted or rendered.

Draft review reports requested capabilities, side effects, credential status, policy findings, and a non-executing dry run. Publication requires the current `draftRevision`, a valid graph and policy review, and every required credential binding. Runs select an immutable pinned or latest-published snapshot.

Expand Down
2 changes: 1 addition & 1 deletion docs-web/content/docs/architecture-node-flows.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Authenticated dashboard routes resolve persisted project ownership from flow, ru
| `webhook_trigger` | Emits secret-authenticated webhook input. |
| `output` | Selects the result. |

These are the executable definitions. Other custom palette concepts remain non-executable until a versioned handler is registered. Graph v1 migration preserves the legacy snapshot and appends deterministic v2.
These are the executable definitions. Other custom palette concepts remain non-executable until a versioned handler is registered. Graph v1 migration preserves the legacy snapshot and appends deterministic v2. Both migrated v1 and canonical v2 graphs are validated as untrusted input: malformed nested members fail closed with stable paths at their original array indices, valid siblings are retained where safe, and repeated validation returns the same ordered issues instead of throwing.

Execution uses immutable publications rather than the mutable editor row. Runs select a pinned publication or the latest published version, then use durable queue claims, leases, bounded quotas, timeout/cancellation propagation, and numbered retry attempts. Expired external attempts with unknown outcomes require operator attention and are not silently replayed. See [Node Flow Durable Execution](/docs/architecture-node-flow-durable-execution).

Expand Down
3 changes: 3 additions & 0 deletions docs/architecture/node-flow-foundation.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,12 @@ Run records are persisted as `NodeFlowRunRecord` and `NodeFlowNodeRunRecord`. Bo
- the graph is acyclic and has a deterministic execution order
- widget fields have required id, label, type, and select option metadata
- widget default values are JSON-safe and match the field type
- malformed v1 and v2 node, edge, port, credential-binding, definition-reference, capability, policy, schema, and metadata members produce stable issues at their original paths

Invalid graphs throw a `ValidationError` with field-level details when persistence is attempted. The validation route returns the same structured issue list without writing data.

Normalization is fail closed and entry oriented. It retains safe siblings, does not renumber issue paths after rejecting an earlier array member, and returns issues in deterministic traversal order so repeated validation is stable.

## Runtime

`src/services/node-flow-runtime-service.ts` executes the current flow version after revalidating the graph. It creates a parent `execution_invocations` row with `type = "node_flow"`, then runs nodes in the validator's topological order.
Expand Down
4 changes: 3 additions & 1 deletion docs/architecture/node-flows.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ A `NodeFlowGraph` contains:

Validation is owned by `src/domain/node-flows/node-flow-validation.ts`. It normalizes ids, labels, positions, widget defaults, and graph shape; rejects missing node/edge arrays; rejects duplicate node ids; rejects edges that point at missing nodes; requires at least one node; and rejects cycles. Widget validation supports `text`, `textarea`, `number`, `boolean`, `select`, `json`, `secretRef`, and `keyValue` fields.

Migration and validation treat persisted Graph v1 and canonical Graph v2 as untrusted input. Malformed collection members are rejected at their original index, such as `nodes[1].ports[0]` or `edges[2]`, while structurally valid siblings remain available to the rest of normalization. Definition references, credential bindings, capabilities, policies, port and graph schemas, and JSON metadata emit deterministic field-level issues instead of throwing. Revalidating the same graph produces the same ordered issue list.

Validation requires every node's type/version reference to resolve through the registry and rejects unknown definitions. Runtime execution then dispatches according to the registered definition's executable state and execution kind; a planning concept is not runnable merely because it has a string type.

The dashboard uses the same backend-owned Graph v2 record as the runtime. The selected project controls library loading; no project means no flow, credential, publication, or run requests. The versioned registry drives palette entries, typed ports, configuration widgets, credential slots, capabilities, side-effect review, and policies. Draft saves use optimistic `draftRevision` checks and surface conflicts without overwriting the newer record.
Expand Down Expand Up @@ -119,7 +121,7 @@ Graph v2 is the single workflow model used by backend, MCP, runtime, and dashboa

The executable registry contains the original deterministic/provider/HTTP nodes plus `condition`, `switch`, `foreach`, `merge`, `delay`, `approval`, `email_draft`, `email_send`, `execute_subflow`, and `webhook_trigger`. Unregistered custom types remain non-executable.

Backend Graph v1 migration retains the exact prior version and appends deterministic v2. Browser canvas v1 migration returns the untouched legacy snapshot separately from the normalized graph.
Backend Graph v1 migration retains the exact prior version and appends deterministic v2. Invalid legacy members are carried across the migration boundary so validation can report their original paths rather than silently dropping them. Browser canvas v1 migration returns the untouched legacy snapshot separately from the normalized graph.

## Dashboard and security prerequisites

Expand Down
101 changes: 67 additions & 34 deletions src/domain/node-flows/node-flow-migrators.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,58 +13,91 @@ export interface NodeFlowMigrationResult<TLegacy = unknown> {
legacySnapshot: TLegacy | null;
}

const cloneJson = <T>(value: T): T => JSON.parse(JSON.stringify(value)) as T;
function cloneUntrusted<T>(value: T, seen = new WeakMap<object, unknown>()): T {
if (!value || typeof value !== "object") return value;
const cached = seen.get(value);
if (cached !== undefined) return cached as T;
if (Array.isArray(value)) {
const clone: unknown[] = [];
seen.set(value, clone);
value.forEach((entry) => clone.push(cloneUntrusted(entry, seen)));
return clone as T;
}
const clone: Record<string, unknown> = {};
seen.set(value, clone);
Object.entries(value).forEach(([key, entry]) => {
clone[key] = cloneUntrusted(entry, seen);
});
return clone as T;
}

export function migrateNodeFlowGraph(graph: unknown): NodeFlowMigrationResult<NodeFlowGraph> {
if (isRecord(graph) && graph.schemaVersion === NODE_FLOW_SCHEMA_VERSION) {
return { graph: cloneJson(graph as unknown as NodeFlowGraph), migrated: false, legacySnapshot: null };
return { graph: cloneUntrusted(graph) as unknown as NodeFlowGraph, migrated: false, legacySnapshot: null };
}

const legacy = isRecord(graph) ? cloneJson(graph as unknown as NodeFlowGraph) : { nodes: [], edges: [] };
const nodes = Array.isArray(legacy.nodes) ? legacy.nodes.map(migrateNode) : [];
const definitionByNode = new Map(nodes.map((node) => [node.id, resolveLatestNodeDefinition(node.type)]));
const edges = Array.isArray(legacy.edges) ? legacy.edges.map((edge) => ({
...edge,
...(edge.fromHandle ? {} : { fromHandle: definitionByNode.get(edge.fromNodeId)?.ports.find((port) => port.direction === "output")?.id }),
...(edge.toHandle ? {} : { toHandle: definitionByNode.get(edge.toNodeId)?.ports.find((port) => port.direction === "input")?.id }),
})) : [];
const legacy = isRecord(graph) ? cloneUntrusted(graph) : {};
const nodes = Array.isArray(legacy.nodes)
? legacy.nodes.map((node) => isRecord(node) ? migrateNode(node) : node)
: legacy.nodes;
const definitionByNode = new Map(
(Array.isArray(nodes) ? nodes : [])
.filter(isRecord)
.map((node) => [stringValue(node.id), resolveLatestNodeDefinition(stringValue(node.type))]),
);
const edges = Array.isArray(legacy.edges) ? legacy.edges.map((edge) => {
if (!isRecord(edge)) return edge;
const fromNodeId = stringValue(edge.fromNodeId);
const toNodeId = stringValue(edge.toNodeId);
return {
...edge,
...(edge.fromHandle ? {} : { fromHandle: definitionByNode.get(fromNodeId)?.ports.find((port) => port.direction === "output")?.id }),
...(edge.toHandle ? {} : { toHandle: definitionByNode.get(toNodeId)?.ports.find((port) => port.direction === "input")?.id }),
};
}) : legacy.edges;

return {
migrated: true,
legacySnapshot: legacy,
legacySnapshot: legacy as unknown as NodeFlowGraph,
graph: {
schemaVersion: NODE_FLOW_SCHEMA_VERSION,
nodes,
edges,
...(legacy.inputSchema ? { inputSchema: legacy.inputSchema } : {}),
...(legacy.schemas ? { schemas: legacy.schemas } : {}),
...(legacy.metadata ? { metadata: legacy.metadata } : {}),
...(legacy.publication ? { publication: legacy.publication } : {}),
},
...(legacy.inputSchema !== undefined ? { inputSchema: legacy.inputSchema } : {}),
...(legacy.schemas !== undefined ? { schemas: legacy.schemas } : {}),
...(legacy.metadata !== undefined ? { metadata: legacy.metadata } : {}),
...(legacy.publication !== undefined ? { publication: legacy.publication } : {}),
} as unknown as NodeFlowGraph,
};
}

function migrateNode(node: NodeFlowNode): NodeFlowNode {
const definition = node.definition
? resolveLatestNodeDefinition(node.definition.type)
: resolveLatestNodeDefinition(node.type);
function migrateNode(node: Record<string, unknown>): NodeFlowNode {
const definition = isRecord(node.definition)
? resolveLatestNodeDefinition(stringValue(node.definition.type))
: resolveLatestNodeDefinition(stringValue(node.type));
return {
...node,
definition: node.definition ?? { type: node.type, version: definition?.version ?? 1 },
ports: node.ports ?? definition?.ports.map((port) => cloneJson(port)) ?? [],
credentialBindings: node.credentialBindings ?? [],
policy: node.policy ?? (definition ? cloneJson(definition.defaultPolicy) : {}),
capabilities: node.capabilities ?? [...(definition?.capabilities ?? [])],
sideEffect: node.sideEffect ?? definition?.sideEffect ?? "none",
disabled: node.disabled ?? false,
};
...node as unknown as NodeFlowNode,
definition: node.definition !== undefined
? node.definition as NodeFlowNode["definition"]
: { type: stringValue(node.type), version: definition?.version ?? 1 },
ports: node.ports !== undefined
? node.ports as NodeFlowNode["ports"]
: definition?.ports.map((port) => cloneUntrusted(port)) ?? [],
credentialBindings: node.credentialBindings !== undefined ? node.credentialBindings : [],
policy: node.policy !== undefined
? node.policy as NodeFlowNode["policy"]
: definition ? cloneUntrusted(definition.defaultPolicy) : {},
capabilities: node.capabilities !== undefined ? node.capabilities : [...(definition?.capabilities ?? [])],
sideEffect: node.sideEffect !== undefined ? node.sideEffect : definition?.sideEffect ?? "none",
disabled: node.disabled !== undefined ? node.disabled : false,
} as NodeFlowNode;
}

export function migrateNodeCanvasGraphV1(graph: unknown): NodeFlowMigrationResult<unknown> {
if (isRecord(graph) && graph.schemaVersion === NODE_FLOW_SCHEMA_VERSION && Array.isArray(graph.nodes) && graph.nodes.every(isCanonicalNode)) {
return migrateNodeFlowGraph(graph);
}
const legacySnapshot = cloneJson(graph);
const legacySnapshot = cloneUntrusted(graph);
const legacy = isRecord(graph) ? graph : {};
const nodes = Array.isArray(legacy.nodes) ? legacy.nodes.filter(isRecord).map((node): NodeFlowNode => ({
id: stringValue(node.id),
Expand All @@ -87,7 +120,7 @@ export function migrateNodeCanvasGraphV1(graph: unknown): NodeFlowMigrationResul
return {
migrated: true,
legacySnapshot,
graph: { schemaVersion: NODE_FLOW_SCHEMA_VERSION, nodes, edges, metadata: { canvasSelection: jsonValue(legacy.selection) } },
graph: { schemaVersion: NODE_FLOW_SCHEMA_VERSION, nodes, edges, metadata: { canvasSelection: untrustedJsonValue(legacy.selection) } },
};
}

Expand All @@ -98,11 +131,11 @@ function readPorts(value: unknown, direction: "input" | "output") {
}

function canvasNodeData(node: Record<string, unknown>): NodeFlowJsonObject {
const config = Array.isArray(node.config) ? Object.fromEntries(node.config.filter(isRecord).map((entry) => [stringValue(entry.id), jsonValue(entry.value)]).filter(([id]) => id)) : {};
return { config, canvasMetadata: jsonValue(node.metadata) };
const config = Array.isArray(node.config) ? Object.fromEntries(node.config.filter(isRecord).map((entry) => [stringValue(entry.id), untrustedJsonValue(entry.value)]).filter(([id]) => id)) : {};
return { config, canvasMetadata: untrustedJsonValue(node.metadata) };
}

const jsonValue = (value: unknown): NodeFlowJsonValue => JSON.parse(JSON.stringify(value ?? null)) as NodeFlowJsonValue;
const untrustedJsonValue = (value: unknown): NodeFlowJsonValue => cloneUntrusted(value ?? null) as NodeFlowJsonValue;
const endpointValue = (value: unknown, key: string): string => isRecord(value) ? stringValue(value[key]) : "";
const stringValue = (value: unknown): string => typeof value === "string" ? value : "";
const isRecord = (value: unknown): value is Record<string, unknown> => Boolean(value) && typeof value === "object" && !Array.isArray(value);
Expand Down
Loading
Loading