Add Vault JWT local CRE coverage and topology docs by prashantkumar1982 · Pull Request #22048 · smartcontractkit/chainlink

prashantkumar1982 · 2026-04-16T18:50:08Z

What Changed

added Vault JWT auth runtime wiring across the Vault gateway handler, Vault DON handler, OCR2 delegate, and CRE job/deployment config paths
aligned JWT auth with the existing client request-digest flow and now require workflow_owner in authorization_details
updated Vault OCR plugin CRUD responses to canonicalize the returned owner to org_id when VaultOrgIdAsSecretOwnerEnabled is on
added a JWT-enabled local CRE Vault topology on top of workflow-gateway-capabilities-don, documented the flags it turns on, and use the default workflow-gateway-capabilities-don topology for the flags-off path
added auth0 and [CRE.Linking] config to the relevant Vault topologies so tests rely on flag behavior instead of config presence
expanded Bucket B Vault smoke coverage to cover allowlist auth with JWT disabled, JWT auth with JWT enabled, JWT rejection when JWT auth is disabled, and mixed allowlist/JWT CRUD flows
refactored the Vault smoke helpers to share CRUD/auth plumbing, moved pure helper logic out of the main test file, and cleaned up scenario/subtest names to match actual behavior
added a local Vault JWT test harness with a fake JWKS issuer, mock linking service, Docker host-rewrite support for static topologies, and direct use of shared Vault label-encryption helpers
added repo-local Local CRE agent guidance under docs/local-cre/agent-skills/local-cre-e2e/ and linked it from docs/local-cre/index.md

github-actions · 2026-04-16T19:26:48Z

CORA - Pending Reviewers

All codeowners have approved! ✅

For more details, see the full review summary.

github-actions · 2026-04-16T19:27:37Z

I see you updated files related to core. Please run make gocs in the root directory to add a changeset as well as in the text include at least one of the following tags:

#added For any new functionality added.
#breaking_change For any functionality that requires manual action for the node to boot.
#bugfix For bug fixes.
#changed For any change to the existing functionality.
#db_update For any feature that introduces updates to database schema.
#deprecation_notice For any upcoming deprecation functionality.
#internal For changesets that need to be excluded from the final changelog.
#nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
#removed For any functionality/config that is removed.
#updated For any functionality that is updated.
#wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

github-actions · 2026-04-16T19:28:02Z

✅ No conflicts with other open PRs targeting develop

trunk-io · 2026-04-16T19:59:16Z

Failed Test	Failure Summary	Logs
`Test_CCIPProgrammableTokenTransfer_EVM2Sui_BurnMintTokenPool`		Logs ↗︎

_{View Full Report ↗︎ ⋅ Docs}

Tofel · 2026-04-17T09:24:24Z

+
+		return retry.RetryableError(err)
+	})
+	if retryErr != nil {


curious, why is that needed? I have never run into job approval errors. Can you share the ones you encountered? Also, should we retry or is it possible that these errors reveal an actual problem/bug, which should be fixed either in the node or in JD?

Yes this was AI trying something.
Removed it now

Tofel · 2026-04-17T09:30:09Z

+// NewTestJWTIssuer creates a fake issuer with one generated RSA key and starts serving JWKS immediately.
+func NewTestJWTIssuer() (*TestJWTIssuer, error) {
+	return NewTestJWTIssuerOnAddr("0.0.0.0:0")
+}


I think for now we can leave this as-is, but in the future we might want to Dockerise it. Especially if one day these tests will run in k8s clusters (dev, stage, etc) as envisioned by Dev Journeys.

Yes, happy to do it.
I assume you mean LinkingService too?
Also want to understand how exactly?
Do you mean a new container image, and then support it in local cre lifecycle?

Exactly, new container image supported in the local CRE lifecycle similar to how Chip Router is supported:

component defined in https://github.com/smartcontractkit/chainlink-testing-framework/tree/main/framework/components

explicitly declared in the env TOML: https://github.com/smartcontractkit/chainlink/blob/develop/core/scripts/cre/environment/configs/workflow-don-solana.toml#L2-L3

started in environment.go

… into codex/vault-jwt-e2e

Tofel

above all you need make the vault system test faster, currently the JWT-based one takes 2x more time than the next slowest test, i.e. ~10 minutes

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a038867c4a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-04-21T17:30:40Z

+		if expectedWorkflowOwner == "" && orgID == "" {
+			expectedWorkflowOwner = req.Id.Owner
+		}
+		err := EnsureRightLabelOnSecret(publicKey, req.EncryptedValue, expectedWorkflowOwner, orgID)


Gate org-label validation behind OrgID owner setting

validateWriteRequest now always allows ciphertext labels that match orgID, but the Vault plugin still drops orgID when VaultOrgIdAsSecretOwnerEnabled is off (see core/services/ocr2/plugins/vault/plugin.go around orgID = "" before EnsureRightLabelOnSecret). In that configuration, gateway-side validation accepts org-labeled secrets and forwards them, then node/plugin validation rejects the same request later, causing avoidable runtime failures for JWT-enabled deployments that haven’t enabled org-as-owner. Gateway validation should be aligned with the same gate (or enforce the gate when org labels are accepted).

Useful? React with 👍 / 👎.

validateWriteRequest allows orgID or workflowOwner.
We have many tests that use this scenario and are passing.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b656838295

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-04-21T19:18:12Z

+			if isVaultJWTAuthEnabledTopology(topology) {
+				t.Run(jwtSubtestName, func(t *testing.T) {
+					ExecuteVaultMixedAuthTest(t, fixture.TestEnv, fixture.Issuer, fixture.LinkingService)


Avoid concurrent Vault auth subtests on shared fixture

ExecuteVaultAllowListBasedTests contains an inner t.Parallel() subtest, so this t.Run can return before the allowlist flow has finished; starting the JWT subtest immediately afterwards reuses the same fixture.TestEnv and fixture.LinkingService while the allowlist flow may still be running. When parallelEnabled is true, this can make allowlist and JWT CRUD operations race against the same Vault state and produce flaky, order-dependent failures.

Useful? React with 👍 / 👎.

Race against same vault state should be ok

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e991a9559f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-04-21T21:33:51Z

+		runtime.Logger().Info(fmt.Sprintf("Vault secret workflow phase completed: %s", phase.Name),
+			"phaseName", phase.Name,
+			"checkCount", len(phase.Checks),
+		)
+		return fmt.Sprintf("Validated phase %s", phase.Name), nil


Track completed phase before returning from onTrigger

onTrigger returns as soon as the first phase evaluates successfully, so multi-phase workflows can get stuck repeatedly reporting an earlier phase if its checks remain true after state advances. In that case the later phase log ("Vault secret workflow phase completed: <phase>") is never emitted, and callers like waitForVaultWorkflowPhase will time out waiting for later lifecycle phases. Persisting/advancing the current phase (or evaluating only the next expected phase) avoids this starvation.

Useful? React with 👍 / 👎.

prashantkumar1982 · 2026-04-21T21:39:01Z

above all you need make the vault system test faster, currently the JWT-based one takes 2x more time than the next slowest test, i.e. ~10 minutes

Dramatically optimized now. Should be super fast on most recent runs now.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e29f274014

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-04-22T15:58:19Z

 				handlers = append(handlers, newDefaultWebAPICapabilitiesHandler())
 			case GatewayHandlerTypeVault:
-				handlers = append(handlers, newDefaultVaultHandler(g.RequestTimeoutSec))
+				handlers = append(handlers, newDefaultVaultHandler(g.RequestTimeoutSec, svcCfg.Auth0))


Validate Auth0 fields before emitting vault gateway specs

svcCfg.Auth0 is forwarded into the vault handler config without any schema validation, so a service config that sets auth0 but omits issuerURL or audience will still produce a job spec and only fail later at runtime when core/services/gateway/handlers/vault.NewHandler calls NewJWTBasedAuth (which hard-fails on missing issuer/audience). This turns a deterministic config error into a delayed startup failure for gateway jobs; the gateway job builder should reject incomplete Auth0 config up front.

Useful? React with 👍 / 👎.

cl-sonarqube-production · 2026-04-22T16:09:12Z

Quality Gate passed

Issues
2 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

prashantkumar1982 force-pushed the codex/vault-jwt-e2e branch from 31cb602 to 077fcbe Compare April 16, 2026 19:29

Tofel reviewed Apr 17, 2026

View reviewed changes

Comment thread .github/workflows/cre-system-tests.yaml Outdated

Tofel reviewed Apr 17, 2026

View reviewed changes

Comment thread ...cripts/cre/environment/configs/workflow-gateway-capabilities-don-vault-jwt_auth-enabled.toml Outdated

Tofel reviewed Apr 17, 2026

View reviewed changes

Comment thread ...cripts/cre/environment/configs/workflow-gateway-capabilities-don-vault-jwt_auth-enabled.toml Outdated

Tofel reviewed Apr 17, 2026

View reviewed changes

Comment thread system-tests/lib/cre/vault/linking_service.go

prashantkumar1982 force-pushed the codex/vault-jwt-e2e branch 2 times, most recently from 5d4b8ed to 7c9b32a Compare April 17, 2026 18:41

github-advanced-security AI found potential problems Apr 17, 2026

View reviewed changes

Comment thread .github/workflows/sigscanner.yml Fixed

prashantkumar1982 force-pushed the codex/vault-jwt-e2e branch 2 times, most recently from 70d73f2 to 7d1f3a9 Compare April 17, 2026 21:51

prashantkumar1982 added 7 commits April 20, 2026 10:29

[codex] Add Vault JWT local CRE test coverage

65be103

[codex] Fix Vault JWT topology and owner responses

dcf8b01

[codex] Fix Vault config lint

8cb595f

[codex] Trim non-Vault CRE changes

0428e8c

increase test timeout

6438b85

Merge branch 'develop' of https://github.com/smartcontractkit/chainlink…

d883c1b

… into codex/vault-jwt-e2e

[codex] Add disabled Vault topology coverage

e3b6ca0

prashantkumar1982 force-pushed the codex/vault-jwt-e2e branch from 0ca1b0c to e3b6ca0 Compare April 20, 2026 17:31

prashantkumar1982 added 6 commits April 20, 2026 12:08

[codex] Simplify Vault handler wiring

39bb4e2

[codex] Align Vault JWT behavior with topology flags

b3c2302

[codex] Simplify Vault CRE topologies

51fe576

Merge remote-tracking branch 'origin/develop' into codex/vault-jwt-e2e

aa34f0f

[codex] Refactor Vault smoke helpers

6cba4ea

[codex] Fix Vault smoke CI wiring

5614e15

Tofel reviewed Apr 21, 2026

View reviewed changes

Comment thread docs/local-cre/agent-skills/local-cre-e2e/SKILL.md Outdated

Tofel requested changes Apr 21, 2026

View reviewed changes

prashantkumar1982 added 2 commits April 21, 2026 10:19

Merge remote-tracking branch 'origin/develop' into codex/vault-jwt-e2e

01f4475

[codex] Address local CRE review feedback

a038867

chatgpt-codex-connector Bot reviewed Apr 21, 2026

View reviewed changes

prashantkumar1982 added 2 commits April 21, 2026 10:57

[codex] Fix vault linking config in CI

5dc28bd

[codex] Fix vault workflow linking config

b656838

chatgpt-codex-connector Bot reviewed Apr 21, 2026

View reviewed changes

prashantkumar1982 added 5 commits April 21, 2026 12:40

[codex] Speed up vault Bucket B workflow checks

2556d92

[codex] Reuse vault workflows across Bucket B phases

ff7c8d7

Merge remote-tracking branch 'origin/develop' into codex/vault-jwt-e2e

d9e3469

[codex] Remove unused vault workflow helpers

11fa4c0

[codex] Increase go_core_tests runner memory

e991a95

chatgpt-codex-connector Bot reviewed Apr 21, 2026

View reviewed changes

prashantkumar1982 requested a review from Tofel April 21, 2026 21:39

[codex] Move go_core_tests off tmpfs runner

6f8ed93

Tofel reviewed Apr 22, 2026

View reviewed changes

Comment thread core/scripts/cre/environment/configs/workflow-gateway-don-grpc-source.toml Outdated

prashantkumar1982 added 2 commits April 22, 2026 08:26

Merge remote-tracking branch 'origin/develop' into codex/vault-jwt-e2e

867ace2

[codex] Start shared test linking service

0cd33b8

prashantkumar1982 requested a review from Tofel April 22, 2026 15:27

[codex] Fix linking service errname lint

e29f274

chatgpt-codex-connector Bot reviewed Apr 22, 2026

View reviewed changes

Tofel approved these changes Apr 22, 2026

View reviewed changes

cedric-cordenier approved these changes Apr 22, 2026

View reviewed changes

prashantkumar1982 added this pull request to the merge queue Apr 22, 2026

Merged via the queue into develop with commit 444ae28 Apr 22, 2026
213 of 215 checks passed

prashantkumar1982 deleted the codex/vault-jwt-e2e branch April 22, 2026 18:06

Conversation

prashantkumar1982 commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Changed

Uh oh!

github-actions Bot commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CORA - Pending Reviewers

Uh oh!

github-actions Bot commented Apr 16, 2026

Uh oh!

github-actions Bot commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trunk-io Bot commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Tofel Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

prashantkumar1982 Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Tofel Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

prashantkumar1982 Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

Tofel Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Tofel left a comment

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

prashantkumar1982 Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

prashantkumar1982 Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

prashantkumar1982 commented Apr 21, 2026

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

cl-sonarqube-production Bot commented Apr 22, 2026

prashantkumar1982 commented Apr 16, 2026 •

edited

Loading

github-actions Bot commented Apr 16, 2026 •

edited

Loading

github-actions Bot commented Apr 16, 2026 •

edited

Loading

trunk-io Bot commented Apr 16, 2026 •

edited

Loading

Tofel Apr 21, 2026 •

edited

Loading