Intermittent bad_jwt / unrecognized JWT kid on Admin API after ES256 key migration

[manu-mu1]

Since a recent JWT signing-key migration on our project, Admin API calls (admin.createUser, getUserById, updateUserById, deleteUser, and occasionally read-only listUsers) intermittently fail with a JWT verification error. The same operation, performed again later as a fresh attempt with no code or credential change, frequently succeeds. We've already opened a support ticket (SU-429921) but wanted to check whether others have hit this, since Free-plan tickets have no guaranteed response time.

Signing-key context

• Current signing key: ECC (P-256 / ES256)
• Previous signing key: legacy HS256 shared secret
• The migration to the current key happened recently; the previous key has not been revoked

Exact error (safe — no secrets)

What we've ruled out

Not a disabled/invalid API key — we independently tested both the new sb_secret_... key and the legacy service-role (JWT-form) key. In sequential read-only testing, the new sb_secret_... key completed 20/20 listUsers() calls successfully, while the legacy service-role key completed 19/20, with one call failing with the same bad_jwt error. Both key types also produced the identical error during separate end-to-end Admin provisioning cycles.
Not a concurrency/parallelism issue on our end — every reproduction was run fully sequentially (no concurrent requests, file-level parallelism explicitly disabled in our test runner via both configuration and CLI flag). The failure still occurs with zero concurrency.
Not deterministic per-call or per-context — the same test file/operation passes cleanly in isolation multiple times in a row, then fails on a completely different specific call when run again minutes later. Successful calls and identical failed calls are interspersed within the same short run.

Installed client versions

• @supabase/supabase-js: 2.110.8
• @supabase/auth-js: 2.110.8 (bundled)
• @supabase/ssr: 0.12.3
• Supabase CLI: 2.109.1
• Node.js: v24.18.0

Question
Is this a known transitional side effect of the HS256→ES256 signing-key migration (e.g. JWKS propagation lag across Auth service nodes, or some code path still referencing the previous key)? Is there anything we should check or wait for on our end, beyond what our open support ticket (SU-429921) is already covering?

[ISMEG-0x00]

Hi @manu-mu1!

Yes, this is a known edge case during key rotation/migration (HS256 → ES256) in distributed setups.

Here is what is happening and what you can check:

Why this happens
JWKS / Auth Node Cache Desync: Supabase Auth (GoTrue) runs across multiple instances/nodes behind a load balancer. Each instance caches the JWKS public keys with a TTL. Until the cache fully expires across all active nodes, requests routed to a node with a stale cache fail with unrecognized JWT kid.

Service Role JWT vs. API Key: Legacy service-role tokens store the key ID (kid) in their header. When using the old service-role JWT with the newly migrated ES256 active configuration, some auth nodes validate against the new JWKS endpoint while others check local state, causing intermittent failures.

Recommended Steps
Force a Key Refresh / Restart: If you are on a paid project, triggering a manual service restart (or contacting support to flush the GoTrue cache) usually clears stuck JWKS nodes immediately.

Migrate to the New Secret Key (sb_secret_...): As you noticed in your test (20/20 success rate), switching completely away from legacy JWT service-role strings to the new sb_secret_... format avoids relying on legacy kid lookups altogether.

Support Ticket: Since you already opened SU-429921, support can easily manually flush the instance cache or check if a specific Auth node in your cluster is failing key synchronization.

[manu-mu1]

Thank for getting back in touch , Zakaria :)

One update to add:

We have since run a second, independent reproduction, still using the new sb_secret_... key exclusively, with no code, credential, or configuration change from the first test . this time the result was:

• 17 successful requests;
• 3 failed requests;
• failures and successes interleaved within the same sequential run.

For every one of these 20 calls, including the three failures, our instrumentation confirmed the same project hostname, the same Node process, the same Admin-client instance, the same apikey/Authorization fingerprint, and the same sb_secret_... credential format — no legacy three-part service_role JWT left our process, and no user session or custom Authorization override was involved.

So across our two tests: the new secret-key format produced 20/20 once, then 17/20 (3 failures, same bad_jwt: unrecognized JWT kid error) on a separate occasion under otherwise identical conditions. This suggests the new key format reduces exposure but does not eliminate the issue, and that the underlying node/cache desync is intermittent across sessions, not just within a single run.

Could you please:

1. flush the GoTrue/JWKS cache for this hosted project, or restart the affected Auth instances;
2. inspect whether one or more Auth or gateway nodes are out of sync with the project's current signing-key metadata;
3. correlate the successful and failed calls using the UTC timestamps we can provide for both test runs;
4. confirm when the cache flush or Auth-node restart has been completed, so we can perform two consecutive controlled regression runs afterward?

We will not rotate any keys, restart the project, change environment variables, or make code or migration changes while you investigate, since the outbound requests are already confirmed identical and use the new secret-key format in both tests.

[ISMEG-0x00]

Hi @manu-mu1

Thanks for providing those detailed test results and instrumentation metrics!

Just to clarify: I am a fellow developer in the community, so I don't have access to Supabase's internal infrastructure, project logs, or backend servers to restart your instances or flush the JWKS cache.

However, your detailed breakdown is extremely valuable for the official Supabase engineering team handling your support ticket. Here is what I recommend doing right now:

Update Support Ticket (SU-429921): Copy and paste these exact findings including the 17/20 test run details, the UTC timestamps for the 3 failed requests, and your request for a GoTrue cache flush/node restart directly into your open support ticket SU-429921.

Why this helps: Confirming that the issue persists even with the sb_secret_... format proves to Supabase staff that a specific Auth instance or gateway node in your region's cluster is serving stale JWKS metadata across multi-node load balancing.

Only official Supabase support engineers have the administrative access required to force a node sync or flush the GoTrue cache for your project. Adding your timestamped logs to the ticket will give them everything they need to fix it quickly!

[QriMi]

We are seeing the same issue on another hosted Supabase project. Support ticket: SU-430784.

We exclusively use a newly generated sb_secret_... key. No legacy service_role JWT is used, and all requests are executed sequentially without concurrency.

We have already:

• Generated and deployed a new secret key
• Redeployed our frontend and backend
• Restarted the Supabase project
• Waited until the Auth service responded successfully five consecutive times

We then performed two read-only Admin Auth tests:

• Before restarting the Supabase project: 40 of 50 requests succeeded (HTTP 200), while 10 failed (HTTP 403).
• After restarting the Supabase project: 41 of 50 requests succeeded (HTTP 200), while 9 failed (HTTP 403).

Every failure returned the same error:

bad_jwt: unrecognized JWT kid

Generating a new secret key and restarting the project did not resolve the issue. Retries reduce the user-facing impact, but the raw failure rate remains close to 20%.

Could Supabase please inspect the hosted GoTrue/Auth gateway nodes for stale or inconsistent JWKS metadata?

[aantti]

See #48272, #48288 - should be resolved by now.

[itsjerryho]

Facing the same issue on another hosted Supabase project.

Have always been using a sb_secret key before this ES256 migration , and all admin requests were working previously.

We have tried:

Generated and deployed a new secret key
Restarted the Supabase project
Still, we're experiencing this error: invalid JWT: unable to parse or verify signature, token is unverifiable: error while executing keyfunc: unrecognized JWT kid for algorithm ES256

Hope Supabase can fix this issue or provide a working solution for this, thank you!

[aantti]

Should be fixed. Apologies!

[aantti]

@sprsquish

[aantti]

Can you check again please?

[QriMi]

Rechecked twice just now from our production backend.

Both sequential test runs completed successfully:

• First run: 50/50 HTTP 200, 0 HTTP 403
• Second run: 50/50 HTTP 200, 0 HTTP 403

That is 100 consecutive successful Supabase Auth Admin API requests. This appears resolved for our project. Thanks!

[aantti]

Many thanks for confirming - and apologies.