Install redpanda console through operator by pvsune · Pull Request #5758 · redpanda-data/redpanda

pvsune · 2022-08-01T09:43:41Z

Cover letter

Support installing Redpanda Console via the operator.

This PR supports the minimum configuration needed to install Console. The PR is huge to review already; other features (see below) will be done in follow up PRs to help in reviewing.

A minimal Console CR might look like:

apiVersion: redpanda.vectorized.io/v1alpha1
kind: Console
metadata:
  name: my-console
spec:
  # Defaults are set https://github.com/redpanda-data/redpanda/pull/5758/files#diff-1120641da9564909cc77411d89009ba02787493e8bde59d2cf6c56e4007c58c9
  server:
    listenPort: 5000
  schema:
    enabled: true
  clusterKeyRef:
    name: rp-abc1234
    namespace: rp-1-234-5678
  deployment:
    image: vectorized/console:master
  connect:
    enabled: true
    clusters:
    - name: connectors-cluster
      url:  http://rp-1-234-5678.rp-abc1234.svc.cluster.local:8083
      tls:
        enabled: true
        secretKeyRef:
          name: my-console-tls
          namespace: default

This PR supports Console configs listed here. The following configs though will be supported in a follow up PR:

Redpanda which enables Console to get exact Redpanda/Kafka version
Console which enables topic documentation through git
Kafka.Protobuf Kafka.Messagepack

Support for mTLS Kafka API will also be in a follow up PR.

The Console controller will create the following resources:

Kafka Service Account, ACLs
ConfigMap
ServiceAccount
Deployment
Service

And they all will be owned by the Console object. It also adds the following finalizers for cleanup:

consoles.redpanda.vectorized.io/service-account
consoles.redpanda.vectorized.io/acl

The controller implements a ThreadSafeStore which synchronize resources (mostly Secrets for TLS certs) across namespaces. This enables the controller to reference a Cluster in different namespace from the Console CR.

Updating the Console spec should change the ConfigMap mounted to Console Deployment. But K8s doesn't do a hot-reload of the Deployment. In order to recreate the Pods when spec changes, we are creating a new ConfigMap and applying it in the Deployment which should trigger a rollout. ConfigMaps are now set to immutable and old ones are deleted on the Console controller reconciliation.

Most of the controller logic are found at pkg/console directory.

Ref

#2142

emaxerrno · 2022-08-10T05:09:43Z

this is cool!

Let's make sure that the user should be able to pass all console configuration options somehow @weeco has really good product intuition here, i'd make him a required reviewer on this.

vladoschreiner · 2022-08-10T13:26:07Z

I opened the conversation about the missing configuration for the external access. Please resolve before finishing the review.

Don't watch Cluster events because we only need SchemaRegistry URL and it cannot be modified. Requeue if Cluster is not configured instead.

…istryAPIURL().

… in any Namespace.

Don't admit creation, update of Console if: - Referenced Cluster is not found - Console is in different namespace and --allow-console-any-ns=false

- Test that SASL user Secret is created - Other resources created need to be mocked before we can test

- Check that Cluster status for Condition ClusterConfigured is not nil - Check that Cluster ExternalListener is not nil

…tantiating KafkaAdmin.

…ation of ConfigMaps. Check count of attached ConfigMap to the Console before creating a new one

…in client. Need to mock calling Kafka Admin in envtests

Test that: - Secret for SASL is created - ConfigMap is created - Deployment is created - Service is created - Console object has the internal URL in status

Running make test formats the code via crlfmt

CreateACLs, DeleteACLs don't return error Use error Aggregate and specifically check for errors in the result

Cluster domain is set in the operator flag

alenkacz

Last round of small comments, now I am ready to approve :)

alenkacz · 2022-08-26T05:17:05Z

src/go/k8s/controllers/redpanda/console_controller.go

+			// Users shouldn't check logs of operator to know this
+			// Adding Conditions in Console status might not be apt, record Event instead
+			// Alternatively, we can have this validation via Webhook
+			r.EventRecorder.Eventf(


this is actually not true, events are in etcd kubernetes/kubernetes#4432 (comment)

alenkacz · 2022-08-26T05:19:02Z

src/go/k8s/controllers/redpanda/console_controller.go

+	if ex := cluster.ExternalListener(); ex != nil && ex.GetExternal().Subdomain != "" {
+		subdomain = fmt.Sprintf("console.%s", ex.GetExternal().Subdomain)
+	} else {
+		r.EventRecorder.Event(


again I think this will flood events, no? 🤔 I would personally just log this

alenkacz · 2022-08-26T05:21:55Z

src/go/k8s/pkg/console/configmap.go

+}
+
+var (
+	// Currently issuing TLS certs through LetsEncrypt


I am not sure we can state this in a generally available operator when ppl can inject any cert they want, right? 🤔

I think ideally this should be checked if IssuerRef is set since it identifies the cert-manager Issuer used to issue certificate. But this isn't the case in v1 - NodeSecretRef is set but the certificates are signed by LetsEncrypt. Is there a way to get this dynamically? :thinking_face:

alenkacz · 2022-08-26T05:22:54Z

src/go/k8s/pkg/console/console.go

+)
+
+// ConsoleConfig is the config passed to the Redpanda Console app
+type ConsoleConfig struct {


isn't there version of this config inside console so we can just import it as we do for redpanda via rpk?

As discussed, I've added a comment about it - the config types in Console don't have Enterprise fields and we don't support all fields yet

…ory.

alenkacz

If the CI passes - LGTM

pvsune · 2022-08-26T10:36:21Z

This fails on the entrypoint test because I mistakenly pushed this directly to upstream instead of in my fork. As discussed, I will create another PR on my fork, close this, and reference this PR on the new one so we can refer the conversations

github-actions bot added the area/k8s label Aug 1, 2022

pvsune force-pushed the pvsune/rp-console branch 17 times, most recently from 2e1c170 to cf8a364 Compare August 8, 2022 18:41

pvsune marked this pull request as ready for review August 9, 2022 06:58

pvsune requested a review from a team as a code owner August 9, 2022 06:58

pvsune requested review from 0xdiba, RafalKorepta, alenkacz, nicolaferraro and vladoschreiner August 9, 2022 06:58

pvsune force-pushed the pvsune/rp-console branch 2 times, most recently from f726677 to bb4c8ee Compare August 10, 2022 04:46

pvsune requested a review from weeco August 10, 2022 11:59

pvsune added 18 commits August 25, 2022 11:33

operator: Requeue if cluster is not ready/configured.

f498393

Don't watch Cluster events because we only need SchemaRegistry URL and it cannot be modified. Requeue if Cluster is not configured instead.

operator: Add comments to Console API fields.

cef9385

operator: Rename Schema spec to SchemaRegistry.

cfd0695

operator: Rename MetricsNamespace spec to MetricsPrefix.

803e624

operator: Add comment why we prefer internal URL in Cluster.SchemaReg…

992ca88

…istryAPIURL().

operator: Add comment about unsetting ConfigMapRef.

bc39fdf

operator: Add --allow-console-any-ns flag to control creating Console…

5517c9c

… in any Namespace.

operator: Setup webhook for Console.

11c6e2f

Don't admit creation, update of Console if: - Referenced Cluster is not found - Console is in different namespace and --allow-console-any-ns=false

operator: Rename console.IsAnyNamespace to IsAllowedNamespace.

ca819ec

operator: Add envtest for Console.

fb626bf

- Test that SASL user Secret is created - Other resources created need to be mocked before we can test

operator: Check for nils.

a49b9c8

- Check that Cluster status for Condition ClusterConfigured is not nil - Check that Cluster ExternalListener is not nil

operator: Use Cluster superuser instead of PandaProxy superuser in in…

982c418

…tantiating KafkaAdmin.

operator: Only look for SchemaRegistry client certs if mTLS is enabled.

cff62bd

operator: Additional check to make sure controller will not flood cre…

cddc915

…ation of ConfigMaps. Check count of attached ConfigMap to the Console before creating a new one

operator: Add KafkaAdminClientFactory interface to abstract Kafka Adm…

9e17aa2

…in client. Need to mock calling Kafka Admin in envtests

operator: Add more tests for Console

b3f6e7d

Test that: - Secret for SASL is created - ConfigMap is created - Deployment is created - Service is created - Console object has the internal URL in status

operator: Run make fmt.

d5ec6dc

Running make test formats the code via crlfmt

operator: Update kubebuilder PROJECT config.

2c8870e

pvsune force-pushed the pvsune/rp-console branch from 490dae0 to 85ac138 Compare August 25, 2022 03:37

pvsune added 2 commits August 25, 2022 13:04

operator: Check for errors in kadm results.

2b333f5

CreateACLs, DeleteACLs don't return error Use error Aggregate and specifically check for errors in the result

operator: Get cluster domain from flag.

3767640

Cluster domain is set in the operator flag

pvsune force-pushed the pvsune/rp-console branch from 85ac138 to 3767640 Compare August 25, 2022 05:05

alenkacz reviewed Aug 26, 2022

View reviewed changes

operator: Add comment about copying config types from Console reposit…

002a99f

…ory.

alenkacz approved these changes Aug 26, 2022

View reviewed changes

pvsune closed this Aug 26, 2022

pvsune mentioned this pull request Aug 26, 2022

Install redpanda console through operator #6242

Merged

pvsune deleted the pvsune/rp-console branch April 18, 2023 03:05

coderabbitai bot mentioned this pull request Mar 21, 2026

docs (k8s): Add Operator Console CRD deployment docs redpanda-data/docs#1629

Open

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Install redpanda console through operator#5758

Install redpanda console through operator#5758
pvsune wants to merge 51 commits intodevfrom
pvsune/rp-console

pvsune commented Aug 1, 2022 •

edited

Loading

Uh oh!

emaxerrno commented Aug 10, 2022 •

edited

Loading

Uh oh!

vladoschreiner commented Aug 10, 2022

Uh oh!

alenkacz left a comment

Uh oh!

alenkacz Aug 26, 2022

Uh oh!

alenkacz Aug 26, 2022

Uh oh!

alenkacz Aug 26, 2022

Uh oh!

pvsune Aug 26, 2022

Uh oh!

alenkacz Aug 26, 2022

Uh oh!

pvsune Aug 26, 2022

Uh oh!

alenkacz left a comment

Uh oh!

pvsune commented Aug 26, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

pvsune commented Aug 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Cover letter

Ref

Uh oh!

emaxerrno commented Aug 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vladoschreiner commented Aug 10, 2022

Uh oh!

alenkacz left a comment

Choose a reason for hiding this comment

Uh oh!

alenkacz Aug 26, 2022

Choose a reason for hiding this comment

Uh oh!

alenkacz Aug 26, 2022

Choose a reason for hiding this comment

Uh oh!

alenkacz Aug 26, 2022

Choose a reason for hiding this comment

Uh oh!

pvsune Aug 26, 2022

Choose a reason for hiding this comment

Uh oh!

alenkacz Aug 26, 2022

Choose a reason for hiding this comment

Uh oh!

pvsune Aug 26, 2022

Choose a reason for hiding this comment

Uh oh!

alenkacz left a comment

Choose a reason for hiding this comment

Uh oh!

pvsune commented Aug 26, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

pvsune commented Aug 1, 2022 •

edited

Loading

emaxerrno commented Aug 10, 2022 •

edited

Loading