SQL: OIDC and access management#580
Conversation
✅ Deploy Preview for rp-cloud ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
kbatuigas
force-pushed
the
DOC-1999-document-feature-oxla-oidc-authn-support
branch
from
d22990c to
66401a7
Compare
kbatuigas
force-pushed
the
DOC-1999-document-feature-oxla-oidc-authn-support
branch
from
243440f to
da74bd4
Compare
|
|
||
| == Authentication | ||
|
|
||
| Redpanda Cloud preconfigures the SQL engine's OIDC settings when SQL is enabled on the cluster, so you do not need to configure an external identity provider for the SQL engine. The engine validates bearer tokens minted by Redpanda Cloud, regardless of whether you sign in to Redpanda Cloud with email and password or SSO. |
There was a problem hiding this comment.
For SME: please confirm if this is correct and relevant
|
|
||
| == Syntax | ||
|
|
||
| === Grant on a table |
There was a problem hiding this comment.
Should this be removed? It appears that this just targets "native" tables. Is the correct syntax for RP tables supposed to be like this example instead? Or is it GRANT SELECT ON TABLE default_redpanda_catalog=>'orders' TO "alice@example.com";?
Feediver1
left a comment
Feediver1
left a comment
There was a problem hiding this comment.
PR Review: SQL: OIDC and access management (#580)
Files reviewed: 7 new .adoc files + nav update (621 additions, 0 deletions)
Overall assessment: The final piece of the SQL GA series. paulzhang97's three engineer concerns all addressed in the current diff. Same broken-sibling-xref / no-What's-New pattern as the rest of the series. Two outstanding SME confirmations and four // TODO markers warrant a quick triage before merge.
What this PR does
Adds the user, authentication, and access-management surface for Redpanda SQL on the rp-sql integration branch:
modules/sql/pages/connect-to-sql/authenticate.adoc(new, 155 lines) — how-to: connect to Redpanda SQL with an OIDC bearer token, OIDC client credentials, or a SCRAM password for legacy clients. Includespsql+psycopg2examples and TLS / reconnect-on-token-expiry guidance.modules/sql/pages/manage/manage-access.adoc(new, 110 lines) — concept: separates authentication (OIDC by default) from authorization (SQL: ManagevsSQL: Accessdata-plane RBAC roles + per-userGRANT/REVOKEagainst catalogs and tables), and explains why Kafka ACLs don't gate query-time access.modules/reference/pages/sql/sql-statements/create-user.adoc,alter-user.adoc,drop-user.adoc(3 new files, 147 lines combined) — SQL statement references for user lifecycle.modules/reference/pages/sql/sql-statements/grant.adoc,revoke.adoc(2 new files, 202 lines combined) — privilege management for tables, external sources (catalogs and storage), and schemas.modules/ROOT/nav.adoc— wires the new pages into the nav tree.
Jira ticket alignment
Ticket: DOC-1999 — "Document feature Oxla OIDC authn support" (extracted from branch name).
Status: Covers everything the ticket implies and goes beyond OIDC alone — adds the full authentication-mode taxonomy (bearer / client credentials / SCRAM-for-legacy), the authorization model (RBAC roles + SQL grants), and the user-lifecycle and grant/revoke statement references. Paulzhang97's three engineering corrections are all reflected in the current diff. ✓
Critical issues (must fix)
-
Two broken xrefs to sibling-PR targets (verified missing on
rp-sql):File:line xref target Provided by authenticate.adoc:17sql:get-started/deploy-sql-cluster.adoc[Enable Redpanda SQL](Prerequisites)PR #571 (still OPEN) authenticate.adoc:50sql:get-started/sql-quickstart.adoc[Quickstart](bearer-token section)PR #571 (still OPEN) -
Missing What's New entry. Fourth PR in the SQL GA series with no entry in
modules/get-started/pages/whats-new-cloud.adoc. As noted across the series, a single coordinated "Redpanda SQL: General availability" entry should announce the whole release.
Suggestions (should consider)
-
Four
// TODOmarkers in the source flag unresolved SME questions. They won't render (AsciiDoc comments), but two of them are content-blocking risk if left unanswered before GA publish:File:line TODO Recommended action manage-access.adoc:28"Confirm validated IdP coverage with engineering (qa-questions.md #16). Add an 'IdPs validated for Redpanda SQL' section listing the providers engineering has explicitly tested (Keycloak is confirmed; Okta, Microsoft Entra ID, Auth0, and Google have not yet been confirmed for SQL)." Resolve before publish — readers will ask "is my IdP supported?" authenticate.adoc:82"OXLA-9382 (concurrent first-time login via connection pool produces duplicate role entries in the distributed catalog) is marked Done in Jira but has no fixVersions set. Confirm with SME the fix is in the GA 2026-05-22 build before publishing this note. If confirmed in GA, delete this comment. If NOT confirmed, restore the user-facing NOTE below." Resolve before publish — if the bug ships in GA, the NOTE block (already drafted in the comment) needs to render to users. authenticate.adoc:110"Reconcile token-endpoint timeout. The RFC says 5s but oxla/src/net/http/http_client.hdefaults to 10s."Lower priority — internal accuracy detail. Decide before GA whether 5s or 10s is authoritative and write the correct value. authenticate.adoc:146"Confirm the correct cross-link for the cluster SSL mode setting in Redpanda Cloud." Lower priority — the surrounding sentence reads fine without it, but the cross-link adds value. -
Two unresolved inline review threads from Kat to SMEs:
manage-access.adoc:24— "For SME: please confirm if this is correct and relevant." This is the Authentication paragraph at line 24 of the current file ("Redpanda Cloud preconfigures the SQL engine's OIDC settings when SQL is enabled on the cluster…"). Needs SME sign-off that the auto-configuration claim is accurate.grant.adoc:38— Kat asks whether theON TABLE table_nameform (Grant on a table) is for native SQL tables only, with Redpanda topic grants needing theON EXTERNAL SOURCE catalog=>patternform instead. The current grant.adoc keeps both forms documented under separate=== Grant on a tableand=== Grant on an external sourcesubheadings, which is the right shape if both are valid — but the SME confirmation is still pending.
-
glossterm:use is inconsistent across the two main content pages:-
authenticate.adocusesglossterm:OpenID Connect (OIDC)[](line 6) andglossterm:bearer token[](line 42) — good. -
manage-access.adocmentions OIDC, RBAC, and ACLs (all on first mention in this page's scope) withoutglossterm:. -
Suggested: Add
glossterm:on the first OIDC mention inmanage-access.adoc:11for consistency. RBAC and ACL on first mention are weaker candidates — the team's glossterm list is what should govern.
-
-
Em-dash check is clean for rendered content. One em dash exists in
authenticate.adoc:111, but it's inside a// TODOcomment quoting RFC text, so it doesn't render. Skip.
Impact on other files
modules/ROOT/nav.adoc✓ — already includes all the new pages.modules/get-started/pages/whats-new-cloud.adoc❌ — no SQL GA entry (Critical #2).- Cross-component xrefs verified:
xref:security:authorization/rbac/rbac_dp.adoc✓xref:sql:connect-to-sql/index.adoc✓- All intra-PR xrefs (between manage-access, authenticate, grant, revoke, user lifecycle pages) ✓
- Only the two
xref:sql:get-started/*references above are unresolved.
- Cross-page consistency with sibling PRs:
manage-access.adocdescribes the SQL: Manage / SQL: Access split thatquery-streaming-topics.adoc(PR #574) refers to in its prerequisites — wording is consistent.- The
default_redpanda_catalog=>'orders'syntax used in grant examples here matches what's used inquery-iceberg-topics.adoc(PR #575) andquery-streaming-topics.adoc(PR #574). ✓ ALTER USER ... WITH PASSWORDfor legacy clients is documented in this PR'salter-user.adocand used inauthenticate.adoc's SCRAM section — consistent.
CodeRabbit findings worth considering
None. CodeRabbit's check passed with no actionable findings.
Outstanding review activity (not findings — just status)
- paulzhang97's three engineering corrections (May 19) are all reflected in the current diff:
auth_method=bearernow correctly written as'-c auth_method=bearer'. ✓auth_method=client_secretnow'-c auth_method=client_secret'. ✓oidc.require_tlsbullet removed. ✓
- No formal APPROVED review yet — reviewDecision is empty. With paulzhang97's concerns addressed, the PR is awaiting a follow-up sign-off.
What works well
- Clean separation of concerns:
authenticate.adocis client-side (how a user/client connects),manage-access.adocis admin-side (how an admin grants access). Each page links to the other. - "How queries reach the underlying topics" section in
manage-access.adocis genuinely valuable — it explains why Kafka ACLs don't gate query-time reads (single internal SASL credential for the catalog), so admins can reason about the security boundary correctly. - Three-mode authentication table in
authenticate.adocsuccinctly distinguishes bearer / client-credentials / SCRAM with concrete "when to use" guidance. - Multiple language-client examples for bearer-token connections —
psqlshell + Pythonpsycopg2. The Python example explicitly notes thatcurrent_userresolves to the token principal, not"ignored". ALTER USER ... WITH PASSWORDpersistence behavior is correctly documented: the Cloud user reconciler doesn't reset passwords on its own, but does revokeCONNECTif the data-plane role is removed.- "Reconnect when a token expires" section addresses the OIDC short-lived-token operational reality — sessions persist past token expiry, but new connections need a fresh token.
grant.adoccleanly separatesON TABLEvsON EXTERNAL SOURCEforms with worked examples for each, including the wildcard=>'orders_*'and the catalog-level (no-pattern) form, plus the documented "wildcard only at end of pattern" constraint.- "Schema-level privileges" subsection in
manage-access.adoccoversUSAGEandCREATE ON SCHEMA. glossterm:macro used on first mention of OIDC and "bearer token" inauthenticate.adoc. ✓- All H2+ headings in sentence case; H1s in title case or all-caps SQL keywords. ✓
- No em dashes in rendered content.
- All code blocks have explicit language tags (
[source,sql],[source,bash],[source,python]). ✓ - CI fully green and the Netlify preview links cover all the new pages.
Final-pass review via /docs-team-standards:pr-review.
|
@kbatuigas Ping me after you get SME approvals and I will do another pass. |
grzebiel
left a comment
grzebiel
left a comment
There was a problem hiding this comment.
fix the -c that is not present anymore. Rest LGTM
| * A Redpanda glossterm:BYOC[] cluster on AWS with Redpanda SQL enabled. See xref:sql:get-started/deploy-sql-cluster.adoc[Enable Redpanda SQL]. | ||
| * OIDC configured for your Redpanda cluster. See xref:security:authorization/gbac/gbac_dp.adoc[Configure GBAC in the Data Plane] for the cluster-level OIDC setup. Configuring OIDC is not specific to Redpanda SQL. | ||
| * https://www.postgresql.org/download/[`psql`^] or another PostgreSQL-compatible client. | ||
| * *A SQL user that matches the principal in your OIDC token.* Redpanda SQL does not auto-create users on first OIDC login. The cluster admin must run `CREATE USER <token-principal>` for each user, where `<token-principal>` is the value of the claim configured by the cluster's `oidc.oidc_principal_mapping` (default `$.sub`, often overridden to `$.preferred_username` for human-readable usernames). Usernames listed in the cluster's `oidc.protected_users` (default `["oxla"]`) cannot authenticate via OIDC even when a token resolves to that name. |
There was a problem hiding this comment.
But our cloud operator does that autmaticaly. The requirement is that user is given access to oxla via RBAC
| If you omit `options`, Redpanda SQL defaults to SCRAM password authentication. Redpanda SQL treats a glossterm:bearer token[] in the password field without `auth_method=bearer` as a SCRAM password, and the connection fails. | ||
|
|
||
| The token must include the standard OIDC claims (`sub`, `aud`, `iss`, `exp`) and must use one of these signing algorithms: `RS256`, `RS384`, `RS512`, or `ES256`. Redpanda SQL rejects tokens signed with `alg: none`. | ||
|
|
||
| // TODO: Confirm the validated IdP coverage with engineering (qa-questions.md #16). | ||
| // E2E tests in oxla currently use Keycloak (tests/blackbox/configurations/keycloak/); | ||
| // Okta, Microsoft Entra ID, Auth0, and Google have not been confirmed as explicitly | ||
| // tested for Redpanda SQL bearer-token auth. List validated IdPs here once confirmed. | ||
|
|
||
| // TODO: Verify aud-as-array + azp validation semantics. The RFC states that when | ||
| // `aud` is an array, `azp` is also validated, but the JWT validator code has not | ||
| // been audited line-by-line for this case. Confirm with SME before publishing. | ||
|
|
There was a problem hiding this comment.
Do we want to describe this? The cloud setup is configured correctly, client is not exposed to that options.
| [source,bash] | ||
| ---- | ||
| PGPASSWORD="$BEARER_TOKEN" \ | ||
| psql "host=<sql-host> port=<sql-port> dbname=oxla user=ignored options='auth_method=bearer' sslmode=require" | ||
| ---- |
There was a problem hiding this comment.
shall we include here the rpk command to acquire the token? PGPASSWORD=$(rpk cloud auth token)
| ==== | ||
| *Concurrent first-time logins.* First-time logins through a connection pool can occasionally produce duplicate role entries in the distributed catalog, after which DDL operations against the role fail. Track https://redpandadata.atlassian.net/browse/OXLA-9382[OXLA-9382^]. As a workaround, have each user log in once with a single connection before using a connection pool against that account. | ||
| ==== |
There was a problem hiding this comment.
this is not an issue since the logic of replicating users to oxla is moved to cloud operator
| // TODO: Confirm the exact token-endpoint timeout to document. | ||
| // Source default in oxla/src/net/http/http_client.h is 10s (request_timeout{10000}), | ||
| // constructed via default-initialised HttpClientConfig in | ||
| // oxla/src/network/leader_factory.cpp:15. The RFC quotes 5s. Document the correct | ||
| // value once verified with SME. |
There was a problem hiding this comment.
This is implementation detail. Doesn't touch user at all
| dbname="oxla", | ||
| user="ignored", | ||
| password=os.environ["BEARER_TOKEN"], | ||
| options="-c auth_method=bearer", |
There was a problem hiding this comment.
| options="-c auth_method=bearer", | |
| options="auth_method=bearer", |
| .psql | ||
| [source,bash] | ||
| ---- |
There was a problem hiding this comment.
should we also add rpk command for acquiring the token here?
|
|
||
| Use client-credentials authentication for service-to-service connections: scheduled jobs, BI tools, or other automated clients that can't perform interactive sign-in. Each Redpanda Cloud service account has a client ID and client secret; Redpanda SQL exchanges those credentials for a token against Redpanda Cloud's token endpoint and validates it. | ||
|
|
||
| Set `options='-c auth_method=client_secret'`, pass the Redpanda Cloud service account's client ID in `user=`, and the client secret in the password field. The resolved principal becomes the session identity. |
There was a problem hiding this comment.
| Set `options='-c auth_method=client_secret'`, pass the Redpanda Cloud service account's client ID in `user=`, and the client secret in the password field. The resolved principal becomes the session identity. | |
| Set `options='auth_method=client_secret'`, pass the Redpanda Cloud service account's client ID in `user=`, and the client secret in the password field. The resolved principal becomes the session identity. |
| [source,bash] | ||
| ---- | ||
| PGPASSWORD="$CLIENT_SECRET" \ | ||
| psql "host=<sql-host> port=<sql-port> dbname=oxla user=<client-id> options='-c auth_method=client_secret' sslmode=require" |
There was a problem hiding this comment.
| psql "host=<sql-host> port=<sql-port> dbname=oxla user=<client-id> options='-c auth_method=client_secret' sslmode=require" | |
| psql "host=<sql-host> port=<sql-port> dbname=oxla user=<client-id> options='auth_method=client_secret' sslmode=require" |
|
|
||
| == Authentication | ||
|
|
||
| Redpanda Cloud preconfigures the SQL engine's OIDC settings when SQL is enabled on the cluster, so you do not need to configure an external identity provider for the SQL engine. The engine validates bearer tokens minted by Redpanda Cloud, regardless of whether you sign in to Redpanda Cloud with email and password or SSO. |
|
|
||
| == Reconnect when a token expires | ||
|
|
||
| OIDC bearer tokens are short-lived. Redpanda SQL validates the token once at connection time and does not re-validate it over the lifetime of the session. After the token expires, the database session continues until it closes for some other reason, but new connections must present a fresh token. Run `rpk cloud auth token` to obtain a new token, then reconnect. |
There was a problem hiding this comment.
@kbatuigas It seems like we need more explanation here about exactly how this works in RP Cloud. At least, how long are the tokens we hand out good for? Can you configure expiry? Needs a bit more info. cc @micheleRP @grzebiel
| :learning-objective-2: Choose the right privilege level when granting access to a Redpanda topic, catalog, or schema | ||
| :learning-objective-3: Recognize how grants on Redpanda catalogs differ from grants on native SQL tables | ||
|
|
||
| Redpanda SQL separates authentication and authorization: |
There was a problem hiding this comment.
@kbatuigas this seems quite elementary to the reader. Not sure it is worth including this statement.
How about "Access to Redpanda SQL is controlled through a combination of Authentication and Authorization controls"
There was a problem hiding this comment.
Also seems like a link to the overall cloud [data plane] RBAC guide would make sense here so people can familiarize themselves with that
| // engineering has explicitly tested (Keycloak is confirmed; Okta, Microsoft | ||
| // Entra ID, Auth0, and Google have not yet been confirmed for SQL). | ||
|
|
||
| == How queries reach the underlying topics |
There was a problem hiding this comment.
@kbatuigas since this is a about identity/access
"How queries access the underlying topics" is a better title
|
|
||
| == How queries reach the underlying topics | ||
|
|
||
| All Redpanda SQL queries connect to the underlying Redpanda cluster as a single internal SASL credential associated with the default Redpanda catalog (`default_redpanda_catalog`), regardless of which SQL user issued the query. The internal credential is provisioned automatically when Redpanda SQL is enabled and is not a user-facing role. |
There was a problem hiding this comment.
"All Redpanda SQL queries connect to the underlying Redpanda cluster using a single internal SASL credential associated with the default Redpanda catalog (default_redpanda_catalog), regardless of which SQL user issued the query. The internal credential is provisioned automatically when Redpanda SQL is enabled and is not a user-facing principal."
Minor tweaks above
| + | ||
| [source,sql] | ||
| ---- | ||
| GRANT SELECT ON EXTERNAL SOURCE default_redpanda_catalog => 'orders' TO "alice@example.com"; |
There was a problem hiding this comment.
Pretty surprised to see this called 'EXTERNAL SOURCE' @mmaslankaprv . ITs part of the same cluster and in no way external . This feels like something from the legacy Oxla we should eventually work to remove in the future. FYI @grzebiel
| * `REVOKE` on a non-matching pattern: `REVOKE SELECT ... \=> 'pattern'` errors if the pattern matches no existing grant for the role. The catalog-level form (no pattern) is idempotent and silently no-ops if no grants exist, so cleanup scripts can safely run it. | ||
| * Inspecting current grants: To see which roles have grants on which catalogs and relations, query `information_schema.role_external_relation_grants`. Catalog-level grants also appear as rows in `information_schema.role_table_grants`, where `table_name` is the catalog name. | ||
|
|
||
| === Schema-level privileges for Redpanda catalogs and SQL storage |
There was a problem hiding this comment.
@kbatuigas 'SQL Storage' vocabulary may be weird here, and perhaps unnecessary. Users in BYOC never need use the STORAGE keyword currently in this version (and I think cant). So might want to remove all references to SQL storage / STORAGE for now.
Also, does the SCHEMA entity even really factor into the current BYOC experience. im sure it will eventually but right now its just the 1 local redpanda+iceberg catalogs right?
certainly IIUC CRESTE STORAGE and maybe CREATE CATALOG (referenced here) is disabled for users
cc @grzebiel or @mmaslankaprv WDYT?
5 participants
Description
This pull request introduces comprehensive documentation for Redpanda SQL's user and access management features, including new reference pages for user lifecycle statements and privilege management, as well as a guide for connecting with OpenID Connect (OIDC). It also updates the navigation to surface these new topics and ensure users can easily find information about authentication and authorization.
New user and access management documentation:
CREATE USER,ALTER USER, andDROP USERSQL statements, detailing syntax, usage, and examples for managing Redpanda SQL users. [1] [2] [3]GRANTandREVOKESQL statements, covering privilege assignment and removal on various database objects, including external sources and schemas. [1] [2]OIDC authentication documentation:
connect-with-oidc.adoc, describing how to authenticate to Redpanda SQL using OIDC bearer tokens or client credentials, with step-by-step instructions and code examples for multiple clients.Navigation and discoverability improvements:
nav.adocto include links to the new OIDC connection guide, user management, and privilege management topics, making these features easier to find in the documentation. [1] [2]Resolves https://github.com/redpanda-data/documentation-private/issues/
Review deadline: 21 May
Page previews
Redpanda SQL > Connect to RP SQL > Authenticate to Redpanda SQL
Redpanda SQL > Manage RP SQL > Manage Access
Reference > Redpanda SQL Reference > Statements > CREATE USER
Reference > Redpanda SQL Reference > Statements > ALTER USER
Reference > Redpanda SQL Reference > Statements > DROP USER
Reference > Redpanda SQL Reference > Statements > GRANT
Reference > Redpanda SQL Reference > Statements > REVOKE
Checks