[CRE] [5/5] Wire confidential workflow execution into CRE by nadahalli · Pull Request #21642 · smartcontractkit/chainlink

nadahalli · 2026-03-23T17:01:21Z

Context

Part of #21635 (confidential workflow execution). [5/5] in the series.
Final wiring PR.

What this does

Wires PRs 1-4 together:

cre.go: starts confidential relay service when config is enabled
models.go: Attributes field on WorkflowSpec
System tests: ConfidentialRelay feature plugin, workflow registration
with attributes, CompileAndDeployConfidentialWorkflow helper

This branch includes code from PRs 1-4 for compilation. As those merge
into develop, rebasing this branch will shrink the diff to just the
wiring and system test changes.

Dependencies

Depends on #21638, #21639, #21640, #21641 (PRs 1-4). Merge those first.

github-actions · 2026-03-23T17:01:38Z

👋 nadahalli, thanks for creating this pull request!

To help reviewers, please consider creating future PRs as drafts first. This allows you to self-review and make any final changes before notifying the team.

Once you're ready, you can mark it as "Ready for review" to request feedback. Thanks!

github-actions · 2026-03-23T17:01:48Z

CORA - Analysis Skipped

Reason: The number of code owners (4) is less than the minimum required (5) and/or the number of CODEOWNERS entries with changed files (8) is less than the minimum required (2).

github-actions · 2026-03-23T17:03:29Z

✅ No conflicts with other open PRs targeting tejaswi/cw-2-relay-handler

Copilot

Pull request overview

Risk Rating: HIGH

Wires confidential workflow execution into CRE end-to-end (registry attributes → syncer routing → confidential module execution → gateway relay plumbing), with accompanying config, DB migration, and system-test helpers/plugins.

Changes:

Add confidential workflow execution support via WorkflowSpec.Attributes, confidential engine routing, and ConfidentialModule.
Introduce gateway + relay DON plumbing (new gateway handler type + CRE subservice) and related config.
Extend system tests to register workflows with attributes and provide a confidential relay feature/plugin.

Targeted areas for scrupulous human review:

DB migration for workflow_specs_v2.attributes and the ORM upsert path.
Filesystem fetcher path validation changes (security boundary check).
Confidential relay request/attestation verification flow and error handling (gateway handler + relay handler).

Recommended reviewers (per CODEOWNERS):

@smartcontractkit/keystone (core/services/workflows, core/capabilities, deployment/cre)
@smartcontractkit/capabilities-team (core/capabilities)
@smartcontractkit/operations-platform (deployment/cre)

Reviewed changes

Copilot reviewed 40 out of 45 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
system-tests/tests/test-helpers/t_helpers.go	Updates workflow artifact creation/registration, adds attributes + confidential deploy helper; currently removes helpers still used elsewhere.
system-tests/tests/go.mod	Bumps deps to support confidential workflow/attestation work.
system-tests/tests/go.sum	go.sum updates for system-tests/tests module.
system-tests/lib/go.mod	Bumps deps to support confidential workflow/attestation work.
system-tests/lib/go.sum	go.sum updates for system-tests/lib module.
system-tests/lib/cre/workflow/workflow.go	Extends workflow registration to include attributes.
system-tests/lib/cre/types.go	Adds `ConfidentialRelayCapability` flag.
system-tests/lib/cre/features/confidential_relay/confidential_relay.go	New system-test feature plugin wiring confidential relay handler + node TOML overrides.
core/store/migrate/migrations/0295_add_workflow_attributes_column.sql	Adds `attributes` column to `workflow_specs_v2` (migration default needs fix).
core/services/job/models.go	Adds `WorkflowSpec.Attributes` DB field.
core/services/workflows/artifacts/v2/orm.go	Persists `attributes` in workflow spec upsert.
core/services/workflows/syncer/v2/handler.go	Routes confidential workflows to confidential engine creation path.
core/services/workflows/syncer/v2/handler_test.go	Adds tests for confidential vs non-confidential routing + malformed attributes.
core/services/workflows/v2/confidential_module.go	New module delegating execution to `confidential-workflows` capability.
core/services/workflows/v2/confidential_module_test.go	Unit tests for attribute parsing, hashing, and module execution request/response handling.
core/services/workflows/v2/engine.go	Adjusts execution timestamp handling and tracing attributes.
core/services/workflows/v2/capability_executor.go	Changes execution timestamp representation in helper.
core/services/workflows/syncer/v2/fetcher.go	Adapts file fetcher to support HTTP(S) URLs for confidential workflows (path boundary check needs hardening).
core/services/workflows/syncer/fetcher.go	Same fetcher change for non-v2 path (path boundary check needs hardening).
core/services/gateway/handlers/confidentialrelay/handler.go	New gateway handler implementing confidential relay fanout/quorum aggregation and callbacks.
core/services/gateway/handlers/confidentialrelay/handler_test.go	Tests for the new gateway handler behavior (quorum, rate limiting, timeouts, etc.).
core/services/gateway/handlers/confidentialrelay/aggregator.go	Aggregator implementing F+1 quorum by response digest.
core/services/gateway/handler_factory.go	Registers the new `confidential-compute-relay` handler type.
deployment/cre/jobs/pkg/gateway_job.go	Adds gateway job wiring for `confidential-compute-relay` handler type and service name mapping.
core/capabilities/confidentialrelay/service.go	New CRE subservice wrapper that starts the confidential relay handler when enabled.
core/capabilities/confidentialrelay/handler.go	New relay DON gateway-connector handler: attestation validation + proxy to vault/capabilities.
core/capabilities/confidentialrelay/handler_test.go	Unit tests for relay DON handler request processing and lifecycle.
core/services/cre/cre.go	Starts confidential relay service when CRE config enables it.
core/config/cre_config.go	Adds `CREConfidentialRelay` interface to config surface.
core/services/chainlink/config_cre.go	Implements `ConfidentialRelay()` config adapter for CRE config.
core/config/toml/types.go	Adds TOML config struct for `CRE.ConfidentialRelay` and merge logic.
core/services/standardcapabilities/conversions/conversions.go	Adds “mock” mapping for capability ID/command conversions.
core/capabilities/launcher.go	Logger type changes + single-DON capability serving fix + cross-DON capability discovery for capability DONs.
core/capabilities/launcher_test.go	Adds regression test for single-DON capability serving + adjusts mock expectations.
core/services/workflows/models.go	Adds a field on `step` struct (supports workflow execution changes).
core/services/workflows/metering/metering.go	Logger interface change + reduces trace logging to a single debug log.
core/services/workflows/syncer/engine_registry_test.go	Minor test tweak (adds const).
core/scripts/cre/environment/environment/workflow.go	Updates workflow registration call to pass attributes argument.
go.mod	Bumps deps (cbor + chainlink-common + teeattestation + nitrite).
go.sum	go.sum updates for root module.
deployment/go.mod	Bumps deps for deployment module (chainlink-common/protos, cbor, teeattestation, nitrite).
deployment/go.sum	go.sum updates for deployment module.
core/scripts/go.mod	Bumps deps for scripts module (chainlink-common/protos, teeattestation, nitrite).
core/scripts/go.sum	go.sum updates for scripts module.
.changeset/confidential-workflow-execution.md	Adds changelog entry for confidential workflow execution feature.

Comments suppressed due to low confidence (2)

core/services/workflows/syncer/v2/fetcher.go:231

The path containment check uses strings.HasPrefix(fullPath, basePath), which is not a safe filesystem boundary check (e.g., basePath /tmp/base will match /tmp/base2/...). Use filepath.Rel (reject any rel path starting with ..) or ensure fullPath is cleaned/absolute and compare against basePath+string(os.PathSeparator).

		if !strings.HasPrefix(fullPath, basePath) {
			return nil, fmt.Errorf("request URL %s is not within the basePath %s", fullPath, basePath)
		}

core/services/workflows/syncer/fetcher.go:197

The path containment check uses strings.HasPrefix(fullPath, basePath), which is not a safe filesystem boundary check (e.g., basePath /tmp/base will match /tmp/base2/...). Use filepath.Rel (reject any rel path starting with ..) or ensure fullPath is cleaned/absolute and compare against basePath+string(os.PathSeparator).

		if !strings.HasPrefix(fullPath, basePath) {
			return nil, fmt.Errorf("request URL %s is not within the basePath %s", fullPath, basePath)
		}

Copilot · 2026-03-23T17:07:19Z

core/store/migrate/migrations/0295_add_workflow_attributes_column.sql

@@ -0,0 +1,5 @@
+-- +goose Up
+ALTER TABLE workflow_specs_v2 ADD COLUMN attributes bytea DEFAULT '';


This migration sets a bytea column default to a text literal (DEFAULT ''), which will fail on Postgres with a type mismatch. Use an explicit bytea default (e.g., ''::bytea or '\x'::bytea) or make the column nullable with no default and handle empty attributes in application code.

Copilot · 2026-03-23T17:07:20Z

system-tests/tests/test-helpers/t_helpers.go

+	require.IsType(t, &evm.Blockchain{}, blockchains[0], "expected EVM blockchain type")
+	deleteErr := creworkflow.DeleteWithContract(t.Context(), blockchains[0].(*evm.Blockchain).SethClient, workflowRegistryAddress, version, uniqueWorkflowName)


deleteWorkflows indexes blockchains[0] without checking length; if the environment has zero blockchains this will panic before the type assertion. Add a length check (and ideally pick the registry chain explicitly rather than assuming index 0).

Copilot · 2026-03-23T17:07:20Z

system-tests/tests/test-helpers/t_helpers.go

+// CompileAndDeployConfidentialWorkflow compiles a workflow WASM binary, copies it to Docker
+// containers, and registers it with confidential attributes on the workflow registry.
+func CompileAndDeployConfidentialWorkflow[T WorkflowConfig](t *testing.T,
+	testEnv *ttypes.TestEnvironment, testLogger zerolog.Logger, workflowName string,
+	workflowConfig *T, workflowFileLocation string, attributes []byte,
+) string {


UniqueWorkflowName and ParallelEnabled were removed from this helper, but they are still referenced by multiple system tests (e.g., system-tests/tests/smoke/cre/cre_suite_test.go, system-tests/tests/regression/cre/cre_regression_suite_test.go). This will break compilation for the system-tests module unless the call sites are updated or these helpers are reintroduced.

trunk-io · 2026-03-23T18:05:52Z

Failed Test	Failure Summary	Logs
`Test_StratReconciliation_RetriesWithBackoff`		Logs ↗︎

_{View Full Report ↗︎ ⋅ Docs}

vreff · 2026-03-24T11:50:18Z

core/capabilities/confidentialrelay/handler.go

@@ -0,0 +1,484 @@
+package confidentialrelay


isn't this file from a prior PR?

Seems like this PR has not perfectly caught the rest of the chain.

Yes, from #21639 [2/5]. Included here for compilation. Will disappear from this PR's diff after #21639 merges.

Fixed. Rebuilt PR from scratch using the clean PR 1-4 branches. All core-test drift removed.

vreff · 2026-03-24T11:50:50Z

core/capabilities/confidentialrelay/service.go

+		trustedPCRs: trustedPCRs,
+		caRootsPEM:  caRootsPEM,


again, don't really think either of these need to be constructor-level params so long as we have the capRegistry param.

Done. Updated NewService to pull config from capabilities registry. cre.go now calls NewService(wrapper, capRegistry, lggr) with no trustedPCRs/caRootsPEM params.

vreff · 2026-03-24T11:51:24Z

core/capabilities/launcher.go

 type launcher struct {
 	services.StateMachine
-	lggr                logger.SugaredLogger
+	lggr                logger.Logger


Probably don't want to mess with this, here or anywhere else below.

Fixed. Removed logger type changes and allWorkflowDONs fix (redundant with LocalOnly). Only the addRemoteCapabilities fix for capability DONs remains (PR #21640).

vreff · 2026-03-24T11:51:38Z

core/capabilities/launcher.go

 	if w.don2donSharedPeer != nil {
 		donPairs := w.donPairsToUpdate(w.myPeerID, localRegistry)
-		err := w.don2donSharedPeer.UpdateConnectionsByDONs(ctx, donPairs, w.p2pStreamConfig)
+		err := w.don2donSharedPeer.UpdateConnectionsByDONs(ctx, donPairs, defaultStreamConfig)


This is the addRemoteCapabilities fix from #21640 [3/5]. Capability DONs need to discover remote capabilities (relay DON calling vault DON). Will disappear after #21640 merges.

vreff · 2026-03-24T11:52:23Z

core/services/workflows/metering/metering.go

 	balance *balanceStore
 	client  BillingClient
-	lggr    logger.SugaredLogger
+	lggr    logger.Logger


this file shouldn't change

vreff · 2026-03-24T11:52:38Z

core/services/workflows/syncer/engine_registry_test.go

 func TestEngineRegistry(t *testing.T) {
 	var srv services.Service = &fakeService{}

+	const id1 = "foo"


vreff · 2026-03-24T11:52:51Z

core/services/workflows/v2/capability_executor.go

 	*Engine
 	WorkflowExecutionID string
-	ExecutionTimestamp  time.Time
+	ExecutionTimestamp  int64


not sure this should be touched

vreff · 2026-03-24T11:53:13Z

core/services/workflows/v2/engine.go

-			creCtx := contexts.CREValue(ctx)
 			// Tracer is no-op if DebugMode is false
 			ctx, span := e.tracer.Start(ctx, "workflow_execution",
 				trace.WithAttributes(
 					attribute.String("workflow_name", e.cfg.WorkflowName.String()),
 					attribute.String("version", "v2"),
-					attribute.String("org_id", creCtx.Org),
-					attribute.String("owner_id", creCtx.Owner),
-					attribute.String("workflow_id", creCtx.Workflow),


why is any of this changing?

vreff · 2026-03-24T11:53:30Z

core/services/workflows/v2/engine.go

 		}
 		e.metrics.IncrementExecutionIDLegacyCounter(ctx)
 	}
-	trace.SpanFromContext(ctx).SetAttributes(attribute.String("execution_id", executionID))


don't think any bit of this file should change

vreff · 2026-03-24T11:54:14Z

core/services/workflows/models.go

 	workflows.Vertex
 	capability capabilities.ExecutableCapability
 	info       capabilities.CapabilityInfo
+	config     *values.Map


concerning to see a change like this. Why do we need this added to the step struct?

Removed. This was core-test drift, not part of the confidential workflow changes.

The enclave needs an authenticated URL to download WASM binaries from the CRE storage service. BinaryURLResolver resolves the raw on-chain URL into a presigned/ephemeral URL per execution. Nil-safe: falls back to the raw URL when no resolver is set. PR 5/5 (#21642) wires this to the storage service retriever.

cl-sonarqube-production · 2026-04-02T13:11:20Z

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
6.4% Duplication on New Code

See analysis details on SonarQube

Wires PRs 1-4 together: - cre.go: start confidential relay service when gateway connector exists - system tests: ConfidentialRelay feature plugin, workflow registration with attributes, CompileAndDeployConfidentialWorkflow helper This branch includes code from PRs 1-4 for compilation. As those merge into develop, rebasing this branch will shrink the diff to just the wiring and system test changes. Part of #21635

…tead)

…ootsPEM

The enclave needs an authenticated URL to download WASM binaries from the CRE storage service. BinaryURLResolver resolves the raw on-chain URL into a presigned/ephemeral URL per execution. Nil-safe: falls back to the raw URL when no resolver is set. PR 5/5 (#21642) wires this to the storage service retriever.

…CC plugin ref - chainlink-common v0.11.2-0.20260330123353-c1870a5eff52 (teeattestation merged into root) - Pavel review fixes: sendResponseAndCleanup, error sanitization, requestTimeoutSec - CC plugin ref updated to 526a251

…d constructErrorResponse

(cherry picked from commit 7b4af73) (cherry picked from commit 9a721064b9ca7d070231a0031b33d1db82b16eef)

The enclave needs an authenticated URL to download WASM binaries from the CRE storage service. BinaryURLResolver resolves the raw on-chain URL into a presigned/ephemeral URL per execution. Nil-safe: falls back to the raw URL when no resolver is set. PR 5/5 (#21642) wires this to the storage service retriever.

Copilot AI review requested due to automatic review settings March 23, 2026 17:01

nadahalli requested review from a team as code owners March 23, 2026 17:01

product-security-plaid-production bot requested a review from ecPablo March 23, 2026 17:01

nadahalli mentioned this pull request Mar 23, 2026

[CRE] Confidential workflow execution #21635

Open

5 tasks

product-security-plaid-production bot requested review from ettec, se3000 and vyzaldysanchez March 23, 2026 17:01

Copilot started reviewing on behalf of nadahalli March 23, 2026 17:01 View session

Copilot AI reviewed Mar 23, 2026

View reviewed changes

nadahalli requested a review from a team as a code owner March 23, 2026 17:19

nadahalli force-pushed the tejaswi/cw-5-wire-and-tests-v2 branch from f652007 to 571766f Compare March 23, 2026 17:26

nadahalli force-pushed the tejaswi/cw-5-wire-and-tests-v2 branch from 81fce03 to 898b32b Compare March 23, 2026 19:08

vreff reviewed Mar 24, 2026

View reviewed changes

nadahalli force-pushed the tejaswi/cw-5-wire-and-tests-v2 branch from 16e4fc9 to 5cfc03f Compare March 31, 2026 16:24

nadahalli changed the base branch from develop to tejaswi/cw-2-relay-handler March 31, 2026 16:29

nadahalli force-pushed the tejaswi/cw-5-wire-and-tests-v2 branch from 5cfc03f to cd40794 Compare April 1, 2026 11:04

nadahalli force-pushed the tejaswi/cw-2-relay-handler branch from cb65a31 to 00a16e6 Compare April 2, 2026 14:12

nadahalli and others added 18 commits April 2, 2026 16:21

Remove launcher fix (relay DON will be configured as workflow DON ins…

0137d85

…tead)

Sync with latest PR 1/2/4 branches for E2E

94e1c07

Simplify relay feature plugin: only set Enabled, drop TrustedPCRs/CAR…

0da1727

…ootsPEM

Update CC plugin refs to b7f4350

057f203

Fix mock capability on-chain type: ACTION not TRIGGER

057d721

Make BinaryURLResolver injectable via WithBinaryURLResolver option

36dc719

Revert BinaryURLResolver changes

629731a

File fetcher: extract filename from HTTP URLs for local resolution

d5a71f7

Update CC plugin refs to d7848a5

1951ac8

Unify sendResponseAndCleanup to handle both success and error paths

cc77f09

Simplify sendResponseAndCleanup.

21ccbf1

Fix exhaustive lint: restore missing switch cases in recordMetrics an…

7c9f3cb

…d constructErrorResponse

fan out relay requests to don nodes concurrently

69d4f2c

sync 5/5 workflow wiring with rebased 4/5

3c4c50f

use cre settings for relay gateway rate limits

b22f1da

(cherry picked from commit 7b4af73) (cherry picked from commit 9a721064b9ca7d070231a0031b33d1db82b16eef)

nadahalli force-pushed the tejaswi/cw-5-wire-and-tests-v2 branch from 0603953 to b22f1da Compare April 2, 2026 14:26

		@@ -0,0 +1,5 @@
		-- +goose Up
		ALTER TABLE workflow_specs_v2 ADD COLUMN attributes bytea DEFAULT '';

		require.IsType(t, &evm.Blockchain{}, blockchains[0], "expected EVM blockchain type")
		deleteErr := creworkflow.DeleteWithContract(t.Context(), blockchains[0].(*evm.Blockchain).SethClient, workflowRegistryAddress, version, uniqueWorkflowName)

Conversation

nadahalli commented Mar 23, 2026

Context

What this does

Dependencies

Uh oh!

github-actions bot commented Mar 23, 2026

Uh oh!

github-actions bot commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CORA - Analysis Skipped

Uh oh!

github-actions bot commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

trunk-io bot commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cl-sonarqube-production bot commented Apr 2, 2026

Quality Gate passed

Uh oh!

Reviewers

github-actions bot commented Mar 23, 2026 •

edited

Loading

github-actions bot commented Mar 23, 2026 •

edited

Loading

trunk-io bot commented Mar 23, 2026 •

edited

Loading