Managed Shared Supavisor rejects valid custom role credentials after SCRAM verification passes

[sankakurio]

Hi Supabase team,

Support ticket: SU-471666

I’m also posting here because this is a Free-plan project and support responses are not guaranteed.

We are seeing password authentication rejection for a custom PostgreSQL login role through Managed Shared Supavisor.

Environment

• Project region: Northeast Asia (Tokyo), ap-northeast-1
• PostgreSQL: 17.6
• Driver: node-postgres (pg) 8.23.0
• Custom role: lifems_activity_direct_login
• Shared Session pooler
• Port: 5432
• Database: postgres
• TLS certificate and hostname verification enabled

The Shared pooler username follows the documented format:

[ROLE].[PROJECT_REF]

The host also matches the exact Dashboard Connect value.

What we verified

During a controlled diagnostic:

• The custom role had LOGIN.
• rolvaliduntil = infinity.
• Expected memberships and grants were correct.
• A SCRAM verifier was stored in pg_authid.
• The generated plaintext password successfully validated against the stored SCRAM verifier.
• The stored verifier also exactly matched the verifier we submitted to PostgreSQL.
• Password URI encoding was tested.
• The failing Session probe passed credentials to the driver as separate fields, so URI password parsing was not involved.
• Automatic retries: 0.
• No circuit-breaker event was observed.
• No invalid-secret-format event was observed.

Despite this, one Shared Session authentication attempt was rejected with:

PASSWORD_AUTHENTICATION_REJECTED

No SQL was executed because authentication failed.

Relevant UTC log events

These are three log records from exactly one authentication attempt, not three retries:

• 2026-09-12T15:01:27.768464Z — CLIENT_HANDLER
• 2026-09-12T15:01:28.269665Z — DATABASE_HANDLER
• 2026-09-12T15:01:28.269790Z — CLIENT_HANDLER

Bounded search window:

2026-09-12 15:00:00–15:03:00 UTC

Direct connection limitation

We could not test authentication against the Direct database endpoint because the client environment could not reach that endpoint.

So we are NOT claiming that this is definitely a Supavisor cache/propagation issue.

Our current classification is only:

SUPAVISOR_SHARED_AUTH_REJECTION

The key point is:

The generated password validates correctly against the SCRAM verifier stored in PostgreSQL, but Shared Supavisor rejects the same credential.

Rollback / safety

After the diagnostic:

• role restored to NOLOGIN
• password set to NULL
• no Vercel runtime secret remains
• no production data writes occurred
• no deployment occurred
• no release activation occurred
• credential buffers were cleared
• no further authentication retries were made

Questions

Could someone help confirm:

1. Are custom PostgreSQL LOGIN roles currently supported through Managed Shared Supavisor Session and Transaction poolers?
2. Could this indicate an auth-query / SecretChecker / tenant credential issue?
3. After setting a password on a custom role, is there any propagation delay or cache invalidation requirement?
4. Is there a way to determine whether the rejection occurred during client → Supavisor authentication or Supavisor → PostgreSQL authentication?
5. Is a pooler refresh/restart ever required after changing a custom role password?
6. Are there any known current issues with custom-role authentication through Managed Shared Supavisor?

I have a sanitized evidence bundle with the relevant pooler events and diagnostic metadata if that would help.

Thanks.

[GaryAustin1]

1. This is documented in the Supabase docs to be a valid method to login in with supavisor. It is specifically mentioned in the SB Prisma guide to use another role and in another custom roles section.
   2)-
2. Not documented, but you certainly not long enough to be a general issue.
   4 -
3. No.
4. No known issues but it comes up from time to time. Sometimes wrong user name format, sometimes password with special chars and sometimes no resolution in public forum and no feedback on what happened with support.

[sankakurio]

Thanks for the reply — that helps confirm that the custom role approach itself is supported.
We have already ruled out the two common causes you mentioned:
The Shared Pooler username is exactly [ROLE].[PROJECT_REF], with the project ref appended once.
The failing probe passed the password to node-postgres as a separate field, not through URI parsing. We also tested percent-encoding separately.
We additionally verified that the generated plaintext password successfully validates against the SCRAM verifier actually stored in pg_authid, and the role had LOGIN with VALID UNTIL infinity.
Despite that, a single Shared Session pooler attempt returned PASSWORD_AUTHENTICATION_REJECTED.
Have you seen cases where Supavisor’s auth_query / SecretChecker does not immediately see or use the current SCRAM verifier for a newly enabled custom role, even though PostgreSQL itself has the correct verifier?
Unfortunately we could not test the Direct endpoint from this client environment, so we have not yet proven whether direct PostgreSQL authentication with the same credential succeeds.
If you know of any additional role property, pooler-side requirement, or diagnostic step worth checking before we retry authentication, I’d appreciate it.

[somuai]

Here is the architectural explanation for the rejection behavior you observed, mapping directly to the internal authentication mechanics of Supavisor.

1. Are custom PostgreSQL LOGIN roles supported through Managed Shared Supavisor?

Yes. In Supavisor, when require_user is false (the default multi-tenant configuration on Supabase), roles other than the internal manager user are resolved dynamically using auth_query (typically SELECT rolname, rolpassword FROM pg_authid WHERE rolname = $1). Both Session mode (port 5432) and Transaction mode (port 6543) fully support custom roles, provided the username adheres to [ROLE].[PROJECT_REF] and the role has LOGIN permissions.

2 & 4. Distinguishing Client -> Supavisor vs Supavisor -> PostgreSQL Rejection

Your log sequence precisely confirms where the failure occurred:

• 15:01:27.768464Z — CLIENT_HANDLER
• 15:01:28.269665Z — DATABASE_HANDLER (+501 ms)
• 15:01:28.269790Z — CLIENT_HANDLER (+125 µs)

The rejection occurred during the Supavisor -> PostgreSQL backend handshake (DATABASE_HANDLER), NOT between your client and Supavisor.

Here is the internal flow:

1. ClientHandler initiates SCRAM-SHA-256 with your driver.
2. ClientAuthentication.fetch_validation_secrets/3 retrieves the stored SCRAM verifier from pg_authid (via SecretChecker or a fallback auth_query).
3. Your client completes the SCRAM exchange. Because validate_scram_proof/2 succeeds, ClientHandler computes the client key (:crypto.exor(Base.decode64!(client_proof), context.signatures.client)) and stores the resolved credentials in UpstreamAuthentication.put_upstream_auth_secrets/2.
4. If client-to-pooler SCRAM had failed, ClientHandler would have terminated immediately with a wrong password exception without ever dispatching to the backend.
5. Instead, 501 ms later, DbHandler (DATABASE_HANDLER) attempted to establish an upstream connection to PostgreSQL using the derived credentials and was rejected by PostgreSQL with code 28P01.
6. Exactly 125 microseconds later, ClientHandler received the fatal termination event forwarded by DbHandler and closed the client socket.

3 & 6. SecretChecker Propagation Delay and Cache Invalidation Mechanics

This behavior is caused by the interaction between the periodic SecretChecker refresh and upstream credential caching:

• In supabase/supavisor, Supavisor.SecretChecker (lib/supavisor/secret_checker.ex) runs on a polling interval of 15 seconds (@interval :timer.seconds(15) plus jitter).
• Supavisor maintains two separate caching layers in ETS / Cachex:
  1. Validation secrets (ValidationSecrets): used by ClientHandler to verify incoming client connections.
  2. Upstream secrets (UpstreamAuthentication / TenantCache): used by DbHandler to authenticate to the backend database.
• When you execute ALTER ROLE [role] PASSWORD '...', SecretChecker detects the changed verifier in pg_authid and updates the validation cache. However, as cataloged in supabase/supavisor#1142 (Periodic SecretChecker refresh updates validation secrets without clearing upstream secrets), SecretChecker does not immediately invalidate upstream cached secrets.
• Upstream secrets are cleared only when a 28P01 error occurs (DbHandler.handle_authentication_error/2 calling delete_upstream_auth_secrets/1) or on specific client wrong-password flows that are rate-limited to 3 per minute (RefreshLimiter).
• If a probe is dispatched within the 15-20 second window after role creation or credential update, a newly spawned DbHandler can attempt authentication against PostgreSQL using mismatched or un-synchronized upstream credential state.

5. Is a pooler refresh/restart required?

No. A pooler restart or project reboot is not necessary. Once the initial upstream 28P01 occurs, DbHandler triggers invalidate_global and delete_upstream_auth_secrets, flushing the stale material. Subsequent connection attempts immediately re-query the current pg_authid verifier and succeed.

Recommended Operational Adjustments

1. Automation Grace Window: In automated provisioning scripts or CI diagnostics that create a custom role or alter passwords, introduce a 20-second settling window before dispatching pooler probes to allow the SecretChecker loop to stabilize.
2. Transient Retry Handling: For connection probes targeting the shared pooler right after DDL role changes, handle the initial authentication attempt with an immediate single-retry fallback, as the first failed checkout automatically purges the stale upstream cache entry.

[sankakurio]

Thanks a lot for the detailed explanation. This was very helpful.

We tested the timing hypothesis in a separate Supabase test project first:

created/updated the custom LOGIN role credential
verified the generated password against the SCRAM verifier stored in PostgreSQL
waited about 25 seconds before connecting through the Shared Session pooler
the very first authentication attempt succeeded
no retry was required
no DATABASE_HANDLER 28P01 was observed

We then repeated the same procedure in production with a 25-second settling window.

The production Shared Session authentication succeeded on the first attempt, and the Shared Transaction pooler connection on port 6543 also succeeded.

So in our case, adding the settling window resolved the authentication failure we had originally reported.

Your explanation about the SecretChecker / upstream credential cache behavior was consistent with what we observed.

Thank you again for taking the time to explain the internal flow and point us toward the relevant Supavisor behavior.