Cloud: mTLS support for Kafka API #520

kbatuigas · 2024-05-24T13:34:19Z

Description

How to enable mTLS for Kafka API on RP cloud clusters.
Also includes minor edits related to mTLS mentioned on other docs.
Resolves https://github.com/redpanda-data/documentation-private/issues/2345
Review deadline: 24 May 2024

Page previews

https://deploy-preview-520--redpanda-docs-preview.netlify.app/current/deploy/deployment-option/cloud/security/cloud-authentication/#mtls

Checks

New feature
Content gap
Support Follow-up
Small fix (typos, links, copyedits, etc)

netlify · 2024-05-24T13:34:34Z

✅ Deploy Preview for redpanda-docs-preview ready!

Name	Link
🔨 Latest commit	`ad575b7`
🔍 Latest deploy log	https://app.netlify.com/sites/redpanda-docs-preview/deploys/665661f756751d00083487ac
😎 Deploy Preview	https://deploy-preview-520--redpanda-docs-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

kbatuigas · 2024-05-24T13:35:19Z

modules/deploy/pages/deployment-option/cloud/cloud-overview.adoc

@@ -210,7 +210,6 @@ image::shared:byoc_apply.png[cloud_byoc_apply]

 Redpanda Cloud does not support the following self-hosted functionality:

- mTLS


@paulzhang97 @micheleRP want to verify that we can make this change.

Not clear about what self-hosted deployment can do. I will defer the question to @bpraseed.

modules/deploy/pages/deployment-option/cloud/security/cloud-authentication.adoc

kbatuigas · 2024-05-24T13:36:43Z

modules/deploy/pages/deployment-option/cloud/security/cloud-enable-mtls-kafka-api.adoc

@paulzhang97 is there anything we should document regarding rotating certificates? Not sure if we need to provide specific guidance there for users.

Do you mean that rotating trusted client CA certificates or client certificates? I don't think we need to document how to rotate the certs since it is customer's responsibility to rotate the certs. If there is feedback later, we will add then.

kbatuigas · 2024-05-24T13:37:12Z

modules/ROOT/nav.adoc

@@ -74,6 +74,7 @@
 **** xref:deploy:deployment-option/cloud/manage-billing/aws-commit.adoc[Use AWS Commits]
 *** xref:deploy:deployment-option/cloud/security/index.adoc[Security]
 **** xref:deploy:deployment-option/cloud/security/cloud-authentication.adoc[Authentication]
+**** xref:deploy:deployment-option/cloud/security/cloud-enable-mtls-kafka-api.adoc[Enable mTLS for Kafka API]


@paulzhang97 @micheleRP would like to double check if this seems like an appropriate place in the navigation tree for this doc.

I think it is fine

@paulzhang97 This is an authentication option, and should appear as a subtopic under AuthN, no? Makes no sense to me that it is at the same level as AuthN.

Right. It is AuthN. I didn't look at the content under Security/Authentication. Seems that it should be under Service authentication?

@Feediver1 -- Security/Authentication in the nav tree is currently a conceptual doc https://docs.redpanda.com/current/deploy/deployment-option/cloud/security/cloud-authentication/ whereas this new content is a task-oriented topic, so I've added it as its own doc. I don't know if we would actually want to turn Security/Authentication as its own subsection, perhaps I'll defer to @micheleRP ?

I don't have a strong opinion about where it lands right now, since we'll be changing so much soon with the Cloud docs reorg. @paulzhang97 or @Feediver1: would you prefer that this content be incorporated into the Authentication page as a nested section in the Service authentication section or kept as a standalone file but nested under Authentication?

@paulzhang97 I found it odd that this is placed outside the scope of the authN heading/umbrella here--thoughts?

I would like to put it under Service authentication section. It sounds better fit to me.

Merge into the requested page.

modules/deploy/pages/deployment-option/cloud/security/cloud-enable-mtls-kafka-api.adoc

kbatuigas · 2024-05-24T14:05:17Z

modules/deploy/pages/deployment-option/cloud/security/cloud-enable-mtls-kafka-api.adoc

+:description: Use the Cloud API to enable mTLS for Kafka API connections on your Redpanda cluster.
+:page-cloud: true
+
+Redpanda Cloud supports mTLS authentication for the Kafka API. 


@paulzhang97 @bpraseed what could we add here regarding why we added this authentication mechanism and why we might recommend it over SASL (is it SASL/SCRAM to be specific?)?

Yes. It is SASL/SCRAM. Good question! I can't think of any clear advantage of mTLS since our SASL is over TLS communication instead of unencrypted traffic over e.g. plain TCP, other than that the algorithms in authenticating client in mTLS is more secure than username/password based SASL.

micheleRP · 2024-05-24T17:17:55Z

modules/deploy/pages/deployment-option/cloud/security/cloud-enable-mtls-kafka-api.adoc

+* `<ca-certificate-pem>`: A trusted Kafka client CA certificate in PEM format. The `ca_certificates_pem` field accepts a list of certificates.
+* `<principal-mapping-rule>`: Configurable rule for mapping the Common Name of Kafka client certificates to Kafka principals.  
+
+For example, the mapping rule `RULE:.*CN=([^,]+).*/\\$1/` maps the following certificate subject to a principal named `test`:


@kbatuigas: This isn't rendering properly with the *

bpraseed · 2024-05-24T18:24:43Z

modules/deploy/pages/deployment-option/cloud/security/cloud-enable-mtls-kafka-api.adoc

+rpk cluster info --tls-enabled
+----
+
+You should get an error like the following:


Shouldnt this be if TLS is not enabled correctly you will get the below error @paulzhang97 ?

This is the actual error returned by rpk. I think tls: certificate required makes sense. It says certificate is required. It could better if e.g. tls: client certificate required.

Feediver1 · 2024-05-29T15:13:49Z

modules/deploy/pages/deployment-option/cloud/security/cloud-authentication.adoc

+
+. Create a service account in your organization, if you have not already done so. Go to the https://cloud.redpanda.com/clients[Clients^] page in the Redpanda Cloud UI and click *Add client* to create a service account. Enter a name and description.
+. Retrieve the client ID and secret by clicking *Copy ID* and *Copy Secret*. 
+. Obtain an access token by making a `POST` request to `https://auth.prd.cloud.redpanda.com/oauth/token` with the ID and secret in the request body. 


Looks like this link is now broken.

Feediver1

Let's immediately test link for https://deploy-preview-520--redpanda-docs-preview.netlify.app/current/deploy/deployment-option/cloud/security/cloud-enable-mtls-kafka-api/#use-the-cloud-api-to-enable-mtls

Co-authored-by: Paulo Borges <paulohtb@hotmail.com>

kbatuigas added 3 commits May 24, 2024 01:10

Add new mtls doc

23e1303

Update related docs with mTLS mention

8693b06

Control plane API is now in v1beta2

0eeb198

kbatuigas requested a review from a team as a code owner May 24, 2024 13:34

kbatuigas commented May 24, 2024

View reviewed changes

modules/deploy/pages/deployment-option/cloud/security/cloud-authentication.adoc Outdated Show resolved Hide resolved

kbatuigas commented May 24, 2024

View reviewed changes

kbatuigas requested a review from paulzhang97 May 24, 2024 13:37

kbatuigas commented May 24, 2024

View reviewed changes

modules/deploy/pages/deployment-option/cloud/security/cloud-enable-mtls-kafka-api.adoc Outdated Show resolved Hide resolved

kbatuigas commented May 24, 2024

View reviewed changes

paulzhang97 requested a review from bpraseed May 24, 2024 14:46

micheleRP reviewed May 24, 2024

View reviewed changes

bpraseed reviewed May 24, 2024

View reviewed changes

kbatuigas and others added 4 commits May 24, 2024 17:17

Apply suggestions from review

0dc46d9

Unnecessary escape character

a52ab8e

merge pages

778bef6

fix

ad575b7

Feediver1 reviewed May 29, 2024

View reviewed changes

Feediver1 approved these changes May 29, 2024

View reviewed changes

Deflaimun merged commit 7022efb into main May 29, 2024
5 checks passed

Deflaimun deleted the 2345_api-mtls-support branch May 29, 2024 15:33

Deflaimun added a commit that referenced this pull request Jun 5, 2024

Cloud: mTLS support for Kafka API (#520)

bd1becf

Co-authored-by: Paulo Borges <paulohtb@hotmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cloud: mTLS support for Kafka API #520

Cloud: mTLS support for Kafka API #520

kbatuigas commented May 24, 2024 •

edited by Deflaimun

Loading

netlify bot commented May 24, 2024 •

edited

Loading

kbatuigas May 24, 2024

paulzhang97 May 24, 2024

kbatuigas May 24, 2024

paulzhang97 May 24, 2024

kbatuigas May 24, 2024

paulzhang97 May 24, 2024

Feediver1 May 24, 2024

paulzhang97 May 24, 2024

kbatuigas May 24, 2024

micheleRP May 24, 2024

Feediver1 May 28, 2024

paulzhang97 May 28, 2024

Deflaimun May 28, 2024

kbatuigas May 24, 2024 •

edited

Loading

paulzhang97 May 24, 2024

micheleRP May 24, 2024

bpraseed May 24, 2024

paulzhang97 May 24, 2024

Feediver1 May 29, 2024

Feediver1 left a comment

		@@ -210,7 +210,6 @@ image::shared:byoc_apply.png[cloud_byoc_apply]

		Redpanda Cloud does not support the following self-hosted functionality:

		- mTLS

Cloud: mTLS support for Kafka API #520

Cloud: mTLS support for Kafka API #520

Conversation

kbatuigas commented May 24, 2024 • edited by Deflaimun Loading

Description

Page previews

Checks

netlify bot commented May 24, 2024 • edited Loading

✅ Deploy Preview for redpanda-docs-preview ready!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kbatuigas May 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Feediver1 left a comment

Choose a reason for hiding this comment

kbatuigas commented May 24, 2024 •

edited by Deflaimun

Loading

netlify bot commented May 24, 2024 •

edited

Loading

kbatuigas May 24, 2024 •

edited

Loading