security/oidc: Basic support for OIDC with Kafka and HTTP #14378

BenPope · 2023-10-23T20:40:51Z

Add basic support for:

Kafka API: SASL/OAUTHBEARER
HTTP APIs (SR/PP/Admin): OpenID Connect

The principal mapped from the sub claim of the JWT.

Docs

New cluster configuration options:

option	description	default
`oidc_discovery_url`	"The URL pointing to the well-known discovery endpoint for the OIDC provider."	`https://auth.prd.cloud.redpanda.com/.well-known/openid-configuration`
`oidc_token_audience`	"A string representing the intended recipient of the token."	`redpanda`
`oidc_clock_skew_tolerance`	"The amount of seconds to allow for when validating the exp, nbf, and iat claims in the token."	`30s`
`http_authentication`	"A list of supported HTTP authentication mechanisms. `BASIC` and `OIDC` are allowed."	`["BASIC"]`
`sasl_mechanisms`	"A list of supported SASL mechanisms. `SCRAM`, `GSSAPI`, and `OAUTHBEARER` are allowed."	`["SCRAM"]`

Backports Required

Release Notes

Features

Kafka API: Support SASL/OAUTHBEARER
HTTP Proxy: Support OpenID connect
Schema Registry: Support OpenID connect
Admin API: Support OpenID connect

oleiman

Probably benefit from a second set of eyes, but lgtm

src/v/security/oidc_authenticator.h

oleiman · 2023-10-24T00:49:16Z

src/v/security/tests/jwt_test.cc

+  // Incorrect issuer
+  {1695887942,
+   R"({"iss": "wrong", "sub": "subject", "aud": "redpanda", "exp": 1695887942, "iat": 1695887942})",
+   "http://docker-rp-1:8080/realms/demorealm",
+   "redpanda",
+   0s,
+   oidc::errc::jwt_invalid,
+   security::acl_principal{security::principal_type::user, "subject"}},


Case repeated from above?

Case repeated from above?

No?

I'd swear this block is identical to https://github.com/redpanda-data/redpanda/pull/14378/files/a283e4c90fe65b0934f28a9caac443e6b6f24de3..c5d693b9ea208b81235db4006e136aef68824c78#diff-616549e7a1138c0069726fb7284164d6e819f7af93df4fc512868db2a8783d27R330-R336

I'm with @oleiman

I'm with @oleiman

Me too, GitHub UI was being silly

src/v/security/oidc_service.cc

oleiman · 2023-10-24T04:17:08Z

src/v/security/oidc_service.cc

+
+    co_return co_await http::with_client(
+      std::move(client),
+      [req_hdr](auto& client) mutable -> ss::future<ss::sstring> {


[req_hdr = std::move(req_hdr)]?

oleiman · 2023-10-24T04:58:09Z

src/v/security/oidc_service.cc

+              "invalid response from discovery_url: {}, errc: {}",
+              response_body,
+              metadata.error());
+            co_return; // errc::invalid_credentials;


Is the commented code intended as inline documentation or cruft?

Is the commented code intended as inline documentation or cruft?

Looks old,

tests/rptest/tests/redpanda_oauth_test.py

oleiman · 2023-10-24T05:36:30Z

src/v/security/request_auth.cc

+              "Unauthorized", ss::http::reply::status_type::unauthorized);
+        }
+        const auto& superusers = _superusers();
+        auto found = std::find(superusers.begin(), superusers.end(), username);


should username be initialized somewhere on the oidc path?

should username be initialized somewhere on the oidc path?

Good spot. Looks like I've dropped some tests in this area.

This is all different, and fixed.

tests/rptest/tests/redpanda_oauth_test.py

michael-redpanda

looking nice so far!

src/v/security/oidc_authenticator.cc

src/v/security/oidc_service.cc

tests/rptest/tests/redpanda_oauth_test.py

src/v/security/oidc_authenticator.cc

src/v/config/configuration.cc

BenPope · 2023-10-25T22:24:28Z

Changes in force-push

Improve use of result
- Use assume_error when has_error was checked.
Improve oidc::errc
- enum class
- Finer-grained errors
Improve logging
- Remove logging that has no context by leveraging new errc
- Increase log-level for failed auth
- Use std::error_code::message
Improve validation for oidc_discovery_url
Improved error handling for oidc_service::update
kafka/client now supports SASL/OAUTHBEARER for principal propagaton with HTTP REST
SR and PP tests

BenPope · 2023-10-25T22:46:02Z

Changes in force-push

Rebase

BenPope · 2023-10-25T22:46:32Z

Changes in force-push

Improved logging for sasl_authenticator

BenPope · 2023-10-25T22:51:05Z

Changes in force-push

Fix lint error due to rebase

vbotbuildovich · 2023-10-26T00:47:10Z

new failures detected in https://buildkite.com/redpanda/redpanda/builds/39826#018b6935-4362-467c-9095-387837653e59: "rptest.tests.cluster_config_test.ClusterConfigTest.test_valid_settings"

vbotbuildovich · 2023-10-26T00:58:46Z

new failures detected in https://buildkite.com/redpanda/redpanda/builds/39826#018b6943-5fab-42b2-881a-1b6466ffeb6e: "rptest.tests.cluster_config_test.ClusterConfigTest.test_valid_settings"

vbotbuildovich · 2023-10-26T01:05:28Z

ducktape was retried in job https://buildkite.com/redpanda/redpanda/builds/39826#018b6943-5fa3-47f8-8177-f2a13bcc6b49

piyushredpanda · 2023-10-26T05:08:51Z

/ci-repeat 1

BenPope · 2023-10-30T19:34:27Z

Changes in force-push

Rebase (sorry - wasm changes forced it)
Add ada-url to cmake/dependencies.cmake
Introduce security/oidc_url_parser and tests
config now relies on oidc_url_parser instead of Boost.URL
security::oidc:make_reques now relies on oidc_url_parser

dotnwat · 2023-10-30T19:40:11Z

Rebase (sorry - wasm changes forced it)

a hack that i've seen Tyler use recently is to push the non-rebased version with the conflict, then do something like leave a comment which helps convince github to not squash the change history, and then push the rebased version. then the link to the minimal diff is preserved.

dotnwat · 2023-10-30T19:43:45Z

build-oss seems to still have some issues

Signed-off-by: Ben Pope <ben@redpanda.com>

michael-redpanda · 2023-10-30T20:22:29Z

build-oss seems to still have some issues

Looks like it's fixed, great stuff @BenPope

build-oss happy

vbotbuildovich · 2023-10-30T22:12:45Z

ducktape was retried in job https://buildkite.com/redpanda/redpanda/builds/40085#018b8268-0291-4a32-896f-f9e65b4297a6

michael-redpanda

lgtm

dotnwat · 2023-10-31T03:03:55Z

src/v/security/config_rcl.cc

+}
+result<parsed_url> parse_url(std::string_view url_view) {
+    parsed_url result;
+    auto url = ada::parse<ada::url_aggregator>(url_view);


BenPope added area/kafka area/docs area/pandaproxy REST interface for Kafka API area/schema-registry Schema Registry service within Redpanda area/security area/admin-api labels Oct 23, 2023

BenPope self-assigned this Oct 23, 2023

github-actions bot added area/redpanda and removed area/docs labels Oct 23, 2023

BenPope requested review from dotnwat, oleiman and michael-redpanda October 23, 2023 20:41

BenPope marked this pull request as ready for review October 23, 2023 21:01

oleiman reviewed Oct 24, 2023

View reviewed changes

BenPope mentioned this pull request Oct 24, 2023

security/oidc: Initial OIDC Support for Kafka API #13476

Closed

7 tasks

michael-redpanda reviewed Oct 24, 2023

View reviewed changes

BenPope force-pushed the oidc_authenticator branch from 39d70dc to d46bd8a Compare October 25, 2023 22:22

BenPope force-pushed the oidc_authenticator branch 2 times, most recently from b603029 to 7348305 Compare October 25, 2023 22:45

BenPope force-pushed the oidc_authenticator branch from 7348305 to 339f2be Compare October 25, 2023 22:50

BenPope requested review from michael-redpanda and oleiman October 25, 2023 22:51

BenPope dismissed michael-redpanda’s stale review via e42869e October 30, 2023 19:34

BenPope force-pushed the oidc_authenticator branch from f69f183 to e42869e Compare October 30, 2023 19:34

BenPope requested a review from a team as a code owner October 30, 2023 19:34

BenPope requested review from andrewhsu and removed request for a team October 30, 2023 19:34

github-actions bot added the area/build label Oct 30, 2023

BenPope requested review from michael-redpanda and dotnwat October 30, 2023 19:34

BenPope added 10 commits October 30, 2023 19:52

security/oidc: Introduce oidc_service

f6bcde5

Signed-off-by: Ben Pope <ben@redpanda.com>

security/oidc: Wire up service

69726db

Signed-off-by: Ben Pope <ben@redpanda.com>

security/oidc: Move request_auth from utils to security

3c4eb16

Signed-off-by: Ben Pope <ben@redpanda.com>

security/oidc: Tidy keycloak

b21103b

Signed-off-by: Ben Pope <ben@redpanda.com>

security/oidc: Tidy ducktape test

0d011f0

Signed-off-by: Ben Pope <ben@redpanda.com>

security/oidc: Introduce authenticator

17998af

Signed-off-by: Ben Pope <ben@redpanda.com>

security/oidc: Introduce sasl_authenticator

4841092

Signed-off-by: Ben Pope <ben@redpanda.com>

kafka/client: Support OAUTHBEARER

f0d9dea

Signed-off-by: Ben Pope <ben@redpanda.com>

security/oidc: Support oidc for kafka api

300e71b

Signed-off-by: Ben Pope <ben@redpanda.com>

security/oidc: Support oidc for request_auth

16063c9

Signed-off-by: Ben Pope <ben@redpanda.com>

BenPope force-pushed the oidc_authenticator branch from e42869e to 16063c9 Compare October 30, 2023 19:57

michael-redpanda approved these changes Oct 31, 2023

View reviewed changes

dotnwat approved these changes Oct 31, 2023

View reviewed changes

BenPope merged commit 0dfb673 into redpanda-data:dev Oct 31, 2023
24 checks passed

michael-redpanda mentioned this pull request Oct 31, 2023

Integrate kafka api with audit logs #14452

Merged

7 tasks

github-actions bot mentioned this pull request Dec 22, 2023

update redpanda appVersion from v23.2.21 to v23.3.1 redpanda-data/helm-charts#950

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

security/oidc: Basic support for OIDC with Kafka and HTTP #14378

security/oidc: Basic support for OIDC with Kafka and HTTP #14378

BenPope commented Oct 23, 2023 •

edited

Loading

oleiman left a comment

oleiman Oct 24, 2023

BenPope Oct 24, 2023

oleiman Oct 24, 2023

michael-redpanda Oct 24, 2023

BenPope Oct 24, 2023

BenPope Oct 26, 2023

oleiman Oct 24, 2023

BenPope Oct 26, 2023

oleiman Oct 24, 2023

BenPope Oct 24, 2023

BenPope Oct 26, 2023

oleiman Oct 24, 2023

BenPope Oct 24, 2023

BenPope Oct 26, 2023

michael-redpanda left a comment

BenPope commented Oct 25, 2023

BenPope commented Oct 25, 2023

BenPope commented Oct 25, 2023

BenPope commented Oct 25, 2023

vbotbuildovich commented Oct 26, 2023

vbotbuildovich commented Oct 26, 2023

vbotbuildovich commented Oct 26, 2023

piyushredpanda commented Oct 26, 2023

BenPope commented Oct 30, 2023

dotnwat commented Oct 30, 2023

dotnwat commented Oct 30, 2023

michael-redpanda commented Oct 30, 2023

vbotbuildovich commented Oct 30, 2023

michael-redpanda left a comment

dotnwat Oct 31, 2023

security/oidc: Basic support for OIDC with Kafka and HTTP #14378

security/oidc: Basic support for OIDC with Kafka and HTTP #14378

Conversation

BenPope commented Oct 23, 2023 • edited Loading

Docs

Backports Required

Release Notes

Features

oleiman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michael-redpanda left a comment

Choose a reason for hiding this comment

BenPope commented Oct 25, 2023

BenPope commented Oct 25, 2023

BenPope commented Oct 25, 2023

BenPope commented Oct 25, 2023

vbotbuildovich commented Oct 26, 2023

vbotbuildovich commented Oct 26, 2023

vbotbuildovich commented Oct 26, 2023

piyushredpanda commented Oct 26, 2023

BenPope commented Oct 30, 2023

dotnwat commented Oct 30, 2023

dotnwat commented Oct 30, 2023

michael-redpanda commented Oct 30, 2023

vbotbuildovich commented Oct 30, 2023

michael-redpanda left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenPope commented Oct 23, 2023 •

edited

Loading