1.4.11

Enclave measurements for this release can be seen here: https://github.com/smartcontractkit/chainlink-confidential-compute/actions/runs/43241181775

Build provenance

Release branch commit	58113cc55ec7e67b14d661e1a2768bf0a91a49f1 (release/1.4.11, tag v1.4.11)
Cherry-picked fixes	5586fa2..0e6d033 from main (#150, #153, #154, #155, #156)
main commit behind it	c623457faec571c19aeb5c6008e1a866dbcba8a9
Capability go.mod pseudo-version	v1.4.12-0.20261112140630-1e6946c8a03e (base), v0.0.0-20261112140630-1e6946c8a03e (framework, enclave-client)
Enclave image tag	sha-58113cc (nitro-enclave-sha-58113cc, nitro-workflows-sha-58113cc; -amd64/-arm64 variants + manifest list)
K8s plugin image tag	enclave-sha-58113cc, enclave-workflows-sha-58113cc; -amd64/-arm64 variants + manifest list

Highlights

Hotfix: direct Vault read share union (#150, #153, #154, #155, #156)

The confidential-workflows enclave now accepts relay-DON secret results backed by at least F+1 signers that agree on the ciphertext and vault public key, combines differing share sets across those signers, verifies each TDH2 share during decryption, and dedupes shares by node index. This supports direct Vault reads, where each relay node can return a different subset of shares. Each signer contributes at most 128 encrypted shares per secret by default, overridable with the injected maxEncryptedSharesPerRelayResult setting. The hotfix also bumps chainlink-common from f9f63cb6 to 079dfdd5 in every module and moves the E2E chainlink pin to 196850e2, which contains chainlink#23946; the E2E pin is test-only. Relay secrets handling was split into remote_dispatcher_secrets.go with no behavior change (#154). Direct Vault reads remain gated per workflow by PerWorkflow.FeatureVaultGetSecretsDirectlyActivePeriod and require chainlink nodes that include the direct GetSecrets path.

Retry GetSecrets across enclave key changes (#141)

The confidential-workflows enclave now derives the relay callback ID from the secrets request identity, including its enclave key and configuration. Retrying an execution on an enclave with a different key gets a fresh signed result instead of a cached response bound to the previous key; identical requests still use the relay memo. This ships in the enclave only, with no relay-node change. Relay/vault logs and the CRE UI show the derived negative callback ID; the enclave logs map it back to the WASM callback. Re-fetching after a key change consumes another Secrets.CallLimit unit.

Shard and workflow-DON labels on events (#143)

Capability execution events and user logs now populate CreInfo.ShardIndex and WorkflowDonID. The capability plugin gets the shard from core and the consumer workflow DON from request metadata. Older cores that do not supply the shard report zero; existing event fields remain intact.

Duration buckets and host memory telemetry (#142)

Capability duration histograms use explicit buckets from 5 ms to 300 s. The external Nitro host labels confidential HTTP executions request.kind=action and supports ENCLAVE_MEMORY_POLL_INTERVAL (default 5 s, minimum 250 ms). Memory gauges report the extremes between exports rather than only the last sample, so short-lived spikes remain visible; guest CPU utilization keeps a sampling window of at least 5 s. Invalid poll settings prevent host startup. Mixed plugin versions temporarily produce mixed histogram bucket sets, and memory alerts can fire earlier because available-memory gauges now report the window minimum.

Wasmtime 50 and dependency updates (#145)

Updates the remaining chainlink-common consumers to f9f63cb65fce and aligns the E2E Chainlink source/module pins at a536af911b01. The confidential-workflows enclave moves from Wasmtime 47 to 50, gains WASM host instrumentation, and inherits a default limit of 1,000,000 table elements per workflow. Large workflows need validation against the new limit. This does not restore the CRE WASM/HTTP limiter integrations reverted in v1.4.10.

Release publication and staging validation (#139, #144)

Adds per-change staging-validation plans and a release-notes generator. Publishing a GitHub release now attaches PCR measurements from the latest release-branch push run at the exact tagged SHA, which must finish successfully, without rebuilding images. A newer failed run blocks publication even if an older same-SHA run succeeded. Measurement artifacts are retained for 30 days.

Rollout notes

• This hotfix changes enclave code, so PCR0 and PCR2 changed for both apps on both architectures. Any TrustedValues already deployed from this release are stale; update them on-chain and roll the enclave images one at a time, blue/green style.

• Direct Vault reads stay disabled until PerWorkflow.FeatureVaultGetSecretsDirectlyActivePeriod is enabled for a workflow, and require chainlink nodes that include the direct GetSecrets path (chainlink#23946). Roll back by disabling the gate.

• The core capability-plugin bump does not deploy enclaves or activate the Wasmtime and GetSecrets changes. New enclave images require their own PCR authorization, MCMS execution, and rollout validation.

• The HTTP amd64 EIF is 232.7 MiB and fits the unchanged 1 GiB launch admission budget. The HTTP arm64 EIF is 320.7 MiB and does not fit the default 1 GiB launch allocation; ARM runtime readiness at that allocation is not established.

• Staging must validate GetSecrets failover across an enclave key change and the Wasmtime/table-limit change, with at least a day of canary observation before production promotion. Retain rollback measurements during rollout.

PCR measurements

Secure images only. Assets ending in -insecure.json are for testing and must not be used as secure trusted measurements.

confidential-http - amd64

PCR	Value
PCR0	3b35e23d6ec9f63766cfc9909765d2c78bde25e7071cfef41c3b02dd2d309e30492e7fe545be1be2d61248ddec623bbb
PCR1	4b4d5b3661b3efc12920900c80e126e4ce783c522de6c02a2a5bf7af3a2b9327b86776f188e4be1c1c404a129dbda493
PCR2	c0b83f54ca372276d428fac63e1b22382fc015eb670c27f396337f1aab0fb592256f99ae46bca68e6d57e3c198aa121d

confidential-http - arm64

PCR	Value
PCR0	69991d4d2b1179087012157b05e7204ab9d3063d7706a59a54593fae1f75bd9baa601fa40e9249b88e1529a6ae6b9599
PCR1	3b4a7e1b5f13c5a1000b3ed32ef8995ee13e9876329f9bc72650b918329ef9cf4e2e4d1e1e37375dab0ba56ba0974d03
PCR2	9822ddee26f6dc140c9efa2d4287db00bdb5c4c391069f852679992d551c5f9da3ca2a938015e8effe0b6b8a3633824e

confidential-workflows - amd64

PCR	Value
PCR0	36c51cc7ee8730ce9806b4028851837fdd9d11125d623fed4fbd611c53601081e8596dc1715504c075c8febdbb13bde7
PCR1	4b4d5b3661b3efc12920900c80e126e4ce783c522de6c02a2a5bf7af3a2b9327b86776f188e4be1c1c404a129dbda493
PCR2	0ab7df1a623f335bdb6ae72f1add534c2bfe63d1c6dbbd3d75bd70c5e96d05f7dd57c92776d497a003c8a0b3ef342558

confidential-workflows - arm64

PCR	Value
PCR0	36f19becbd93953c8ed529a97c9bbdc4d54962e26ecee7b5d09efcfb715bdab87ab696c28380c4fa79974a3466efe238
PCR1	3b4a7e1b5f13c5a1000b3ed32ef8995ee13e9876329f9bc72650b918329ef9cf4e2e4d1e1e37375dab0ba56ba0974d03
PCR2	75ec126deccf55d0d10d7d6638521b2def07bfe1a11bed11931e206550fb51454c669e43cde11bcfe903e1ecaf93f2a6

Full diff

v1.4.10...v1.4.11

• Staging validated

Staging validation plans

Staging validation

priv-724-732-metric-buckets-request-kind (#142)

Change

Capability plugin: duration histograms forwarded through MetricsEmitter
(capability-side and enclave-forwarded *_duration) use explicit buckets from
5ms to 300s instead of the OTel defaults (0, 5, 10, 25, 50... seconds).
Host: confidential-http executions now carry request.kind="action" on the
host execution metrics, so the Enclave Hosts dashboard's Request Kind variable
covers HTTP. The host's enclave memory poll interval (default 5s) can be set
with ENCLAVE_MEMORY_POLL_INTERVAL (a Go duration of at least 250ms, such as
500ms). The memory gauges now report the extremes since the previous export
(highest Go runtime memory and RSS, lowest available memory) instead of the
latest poll, so fast polling catches spikes between the 10s exports. Guest CPU
utilization is computed over windows of at least 5s regardless of poll rate.

Validation steps

1. After the capability plugin rolls to staging, list the bucket bounds:
   count by (le) (confidential_workflows_1_0_0_alpha_app_execution_completed_duration_bucket{platformEnv="staging-testnet"})
2. On the Confidential Compute dashboard (staging), check App Execution
   Latency, GetSecrets Latency, Signature Verification Latency and Shares
   Combining Latency for both capabilities.
3. After the host image rolls to staging:
   count by (app_id, request_kind) (confidential_compute_enclave_execution_duration_seconds_count{deployment_environment="stage"})
4. On the Enclave Hosts dashboard, select Request Kind = action and confirm
   the HTTP panels populate.
5. Host startup logs show polling enclave memory with interval=5s, or the
   configured value if ENCLAVE_MEMORY_POLL_INTERVAL is set.
6. Set ENCLAVE_MEMORY_POLL_INTERVAL=500ms on one staging deployment, then
   compare its Guest Memory Headroom and Enclave RSS Memory panels with a
   deployment left at the default, and check guest CPU on both:
   confidential_compute_enclave_guest_cpu_utilization_ratio{deployment_environment="stage"}
7. Check resource poll duration on the 500ms deployment:
   histogram_quantile(0.99, sum by (le, k8s_pod_name) (rate(confidential_compute_enclave_resource_poll_duration_seconds_bucket{deployment_environment="stage"}[5m])))

Success criteria

• Step 1 returns le values 0.005 through 300 plus +Inf (21 buckets) for
  series from upgraded nodes.
• Workflow app execution and GetSecrets P99 move off the 10s/25s bucket edges
  and track real latency; Signature Verification and Shares Combining read
  about 5ms or less instead of a flat 0.
• Step 3 shows request_kind="action" for confidential-http@1.0.0-alpha
  alongside the existing workflow kinds.
• Step 6: the 500ms deployment exports a series every 10s as before, and its
  available memory sits at or below the default deployment's under similar
  load. Guest CPU utilization moves smoothly rather than flipping between 0
  and 1.
• Step 7: poll P99 stays under 5ms, and execution latency on the 500ms
  deployment matches the default one.

Edge cases and risks

• While nodes run mixed plugin versions, histogram_quantile sums old and new
  bucket sets, so quantiles can look jagged until every node is upgraded.
• 21 buckets instead of 16 adds about 30% more bucket series per histogram.
• The enclave still rounds durations to 10ms, so sub-5ms steps stay
  indistinguishable from 0; that rounding is intentional and unchanged.
• Adding request.kind to HTTP host series starts new series once (no
  dashboard queries depend on its absence).
• An invalid ENCLAVE_MEMORY_POLL_INTERVAL, or one below 250ms, stops the
  host at startup. Unset keeps the current 5s.
• Memory alerts on available memory see the lowest value per export rather than
  the latest, so they can fire slightly earlier.
• After a failed poll, the extremes gathered earlier in that window are
  exported once before the memory series disappear, so a spike just before an
  enclave crash stays visible.
• Each failed poll logs a warning, so at 250ms an unreachable enclave logs four
  warnings a second.
• Roll back if host export errors or cardinality alerts appear.

bump-chainlink-196850e (#150)

Change

Bumps chainlink-common to 079dfdd5 (#2528, per-capability Confidential
Compute overrides) in every module. The previous pin, 6cbefe46, was the
pre-merge branch commit of the same change and no longer resolves. Also bumps
the chainlink pin for E2E tests to 196850e2, which adds the direct Vault
GetSecrets path (chainlink #23946); that part is test-only.

Validation steps

• Deploy the confidential-workflows and confidential-http enclaves to staging
  and confirm all enclaves come up and reach quorum.
• Watch the existing confidential-workflows and confidential-http canaries for
  at least a day.

Success criteria

• Canaries stay green, with no new execution failures.

Edge cases and risks

• Rollback: redeploy the previous enclave release.

ci/staging-validation-check (#139)

Change

Adds the staging-validation PR check, its make targets, and the release-notes
generator. CI and tooling only; nothing deployed changes.

Validation steps

None in staging. On this PR, confirm the staging-validation check passes with
this file present.

Success criteria

The staging-validation check is green on this PR, and fails on a PR that adds
no staging-validation/*.md file or leaves a required section empty.

Edge cases and risks

• Bot PRs (dependabump) and PRs labelled no-staging-validation skip the check.
• Adding or removing the label re-runs the check (labeled/unlabeled triggers).

devsvcs-6784-shard-metadata (#143)

Change

Confidential workflow events (capability started/finished, user logs) carry shard_index and workflow_don_id in CreInfo. Shipped by the capability plugin (framework); bumps chainlink-common.

Validation steps

Run a confidential workflow on a sharded staging node and inspect the emitted CapabilityExecutionStarted/Finished and WorkflowUserLog events in Beholder.

Success criteria

CreInfo.ShardIndex equals the node's configured shard and WorkflowDonID equals the consumer workflow DON; existing events keep flowing.

Edge cases and risks

Nodes without the core change send shard 0. Additive proto fields only; roll back by reverting the plugin bump.

priv-707-relay-callback-id (#141)

Change

The confidential-workflows enclave sends the relay DON a callback ID for
runtime GetSecrets derived from the request (enclave key, config, secrets),
so a retry on an enclave with a different key gets a fresh result instead of
the one signed for the previous key. Ships in the confidential-workflows
enclave only; no node or relay change.

Validation steps

• Run a confidential workflow that calls GetSecret and confirm it succeeds.
• In the enclave logs, find [remoteDispatcher] derived relay callback ID for secrets request for that execution and note callbackID and the negative
  relayCallbackID.
• In the relay node logs for the same execution, confirm fetched and signed secrets response carries key get-secrets/<workflowID>/<executionID>/<relayCallbackID>.
• Force a key change mid-execution: while the workflow is fetching secrets,
  delete one enclave-workflows-* pod with --grace-period=0 --force, so the
  executor retries the execution on the other replica.
• Watch the existing confidential workflows canaries and the GetSecrets
  success-rate panel on the Confidential Compute dashboard for the soak period.

Success criteria

• Every runtime GetSecrets logs a negative relayCallbackID, and the relay
  memo key uses it.
• The execution retried after the forced pod deletion completes; no
  no relay-DON result reached quorum error with 0 valid signatures appears
  on the GetSecrets path.
• Canaries stay green and the GetSecrets success rate and latency are unchanged.

Edge cases and risks

• Relay and vault logs and the CRE UI get-secrets step show the derived negative
  ID instead of the WASM call number; the enclave mapping log bridges them.
• A re-fetch after a key change is charged again against Secrets.CallLimit
  (default 5 per execution); workflows near the limit can fail on retry.
• Enclaves on the previous version keep the old behavior until replaced.
• Roll back by redeploying the previous enclave image; there is no node-side
  state to undo.

confidential-workflows share union (#156)

Change

For each requested secret, the confidential-workflows enclave accepts relay-DON
results backed by at least F+1 distinct signers that agree on the ciphertext and
vault public key. Shares may differ between signers. The enclave combines the
shares from those signers, and during decryption verifies each TDH2 share and
dedupes by node index.

Only the enclave changes. Direct Vault reads require chainlink nodes that
include the direct GetSecrets path (smartcontractkit/chainlink#23946).

Pre-staging tests

1. Unit tests: go test ./app/ in enclave/apps/confidential-workflows and
   go test ./util/.
2. E2E: make e2e-local-conf-workflows runs
   TestConfidentialWorkflowsEngineE2E, which exercises the OCR path and then
   enables direct reads at runtime for its direct-read phase. The pinned
   chainlink commit must include the direct GetSecrets path.

Validation steps

1. Deploy the enclave with PerWorkflow.FeatureVaultGetSecretsDirectlyActivePeriod
   disabled. Confirm the confidential-workflows canaries stay green and note
   mergedShares in the enclave's relay-DON secrets quorum reached log.
2. Enable PerWorkflow.FeatureVaultGetSecretsDirectlyActivePeriod for the
   staging canary workflows.
3. Trigger cn_confidentialworkflows_a, cn_cw_httpget_a and
   cn_cw_chainwrite_a. Confirm the vault nodes log
   served direct get secrets request. This is a debug-level log.
4. Confirm the enclave logs relay-DON secrets quorum reached with a larger
   mergedShares than in step 1.
5. Disable PerWorkflow.FeatureVaultGetSecretsDirectlyActivePeriod and confirm
   the canaries stay green.

Success criteria

• Confidential-workflows canaries stay at 100% success with direct reads on.
• No no relay-DON result reached quorum,
  relay-DON secrets results disagree on vault public key or
  failed to aggregate decryption shares errors.
• Secret-read latency is better than with direct reads off.

Edge cases and risks

• Enable direct reads only after every enclave serving the workflow DON runs
  this version.
• A faulty relay node can add invalid or undecryptable shares. The enclave
  skips them and decrypts once it has T valid distinct shares.
• During a DKG reshare, relay nodes can disagree on the vault public key. If no
  group reaches F+1 signers, the request fails with
  no relay-DON result reached quorum. If two secrets reach quorum under
  different keys, it fails with
  relay-DON secrets results disagree on vault public key. The enclave does
  not retry either error; the next workflow execution does.
• Each signer contributes at most 128 encrypted shares per secret by default.
  Extra shares are dropped with a truncating oversized relay share list
  warning. Raise the cap with the injected maxEncryptedSharesPerRelayResult
  setting if honest responses exceed it.
• To roll back, disable PerWorkflow.FeatureVaultGetSecretsDirectlyActivePeriod.
  No enclave rollback is needed.

Staging result

Pending.

bump-chainlink-a536af9 (#145)

Change

Bumps chainlink-common from daa34bb8 to f9f63cb6 in the confidential-workflows enclave app, nitro host, config-tracker, and enclave-client. The common range includes wasmtime v50 (#2497), a per-workflow wasm table-element limit (#2466), and wasmtime host instrumentation (#2474). Also bumps the chainlink core pin for E2E tests to a536af9; that part is test-only.

Validation steps

• Deploy the new confidential-workflows and confidential-http enclaves to staging and confirm all enclaves come up and reach quorum.
• Watch the existing confidential-workflows and confidential-http canaries for at least a day.
• On the enclave dashboards, compare execution success rate, /execute p99 latency, and enclave memory against the previous release.
• In Loki, search confidential-workflows enclave logs for wasm/wasmtime errors, especially table-limit or instantiation failures.

Success criteria

• Canaries stay green, with no new execution failures.
• Execution success rate and p99 latency stay within the previous release's range.
• Enclave memory stays in line with the current baseline, with no new growth over pod lifetime.
• No wasm table-limit or wasmtime instantiation errors in enclave logs.

Edge cases and risks

• wasmtime v50 changes the native engine running workflow binaries inside the enclave; watch for compile/instantiate regressions or memory growth.
• The new table-element limit could reject large workflow binaries that ran before. Roll back if real workflows start failing with limit errors.
• Rollback: redeploy the previous enclave release.