Authentication of Schema Registry on K8s

[rudramaniteja]

Hi,

I am using schema registry image "confluentinc/cp-schema-registry:7.0.1" on Kubernetes 1.23. Can you help me to set basic authentication to schema registry. So that when I want to view schema's, First I should provide my credentials to my pod and then I should be able to access.

Does schema registry supports basic auth in kubernetes?

Thanks!

[OneCricketeer]

Kubernetes doesn't limit the features that are available. What have you tried thus far?

[rudramaniteja]

apiVersion: v1
kind: Service
metadata:
  name: schema-registry
  labels:
    app: schema-registry
spec:
  selector:
    app: schema-registry
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8081
---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: schema-registry
  labels:
    app: schema-registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: schema-registry
  template:
    metadata:
      labels:
        app: schema-registry
    spec:
      containers:
      - name: schema-registry
        image: confluentinc/cp-schema-registry:7.1.1
        ports:
        - containerPort: 8081
        imagePullPolicy: Always
        env:
        - name: SCHEMA_REGISTRY_HOST_NAME
          value: schema-registry
        - name: SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS
          value: "kafka.svc:9092"
        - name: compatibilityLevel
          value: NONE
        - name: SCHEMA_REGISTRY_LISTENERS
          value: http://0.0.0.0:8081
        command:
        - bash
        - -c
        - unset SCHEMA_REGISTRY_PORT; /etc/confluent/docker/run

Here added my deployment and service YAML files. I have created an Ingress and service is working as expected. But what I wanted is, It should work with basic authentication. I can set the basic authentication at Ingress. But even If I do a port-forward to the schema-registry pod, It should ask first ask me to get authenticated. Ingress authentication is getting bypassed here. If there is a facility to achieve this either from Kubernetes or from Schema-Registry side, do let me know. Thanks for help!

[OneCricketeer]

Auth can be a added without any external web service.

But, given that Ingress or Kubernetes issues are not specific to this repo, you may get better support at the Confluent Developer Forum. You'll need to mention what Ingress you're actually using, e.g. Nginx or something else?

[rudramaniteja]

I'm using nginx ingress. How auth can be added without any external web services?

[OneCricketeer]

Again, what have you tried? There's a whole documentation section dedicated to this topic.

https://docs.confluent.io/platform/current/schema-registry/security/index.html#configuring-the-rest-api-for-basic-http-authentication

[rudramaniteja]

https://docs.confluent.io/platform/current/schema-registry/security/index.html#configuring-the-rest-api-for-basic-http-authentication

[rudramaniteja]

From this link, I have tried giving jaas file. I was asked for authentication but when I given the username and password its not validating. And I want this to implement on kubernetes. Can you provide a link to try the deployment on kubernetes. Are there any env variables for the same scenario?

[OneCricketeer]

I want this to implement on kubernetes.

I understood that from your original post. As mentioned, there's nothing unique about Kubernetes that needs configured for authentication of the service itself. If you want to use an Ingress to manage authentication, then refer documentation for the Ingress instead...

Are there any env variables for the same scenario?

Everything that can be configured in the Registry server config maps to an environment variable of the registry container, yes. https://docs.confluent.io/platform/current/installation/docker/config-reference.html#sr-long-configuration

The JAAS file/config needs provided in KAFKA_OPTS variable, as mentioned in that page.

Perhaps you'll find what you're looking for at https://github.com/confluentinc/confluent-kubernetes-examples

[rudramaniteja]

Hi,

Thanks for the reply.

From the link, https://docs.confluent.io/platform/current/installation/docker/config-reference.html#sr-long-configuration you provided in the above comments, I don't find variables related to the Basic authentication.

But from the examples, I have tried installing through the helm repo and applied all configuration from this file https://github.com/confluentinc/confluent-kubernetes-examples/blob/4ceeb1a339c2a690aa121395520e650ae06d8fec/security/plaintext-basic-auth-ConnectAndSchemaRegistry/confluent-platform.yaml. Since the cluster is completely deployed by itself, SchemaRegistry is asking me username:password

So my ask is, Can I only deploy this SchemaRegistry and connect with my existing Kafka cluster? If yes, can you please provide me that link/doc/script?

And another approach is through normal deployment I added above..

 env:
        - name: SCHEMA_REGISTRY_HOST_NAME
          value: schema-registry
        - name: SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS
          value: "kafka.svc:9092"
        - name: compatibilityLevel
          value: NONE
        - name: SCHEMA_REGISTRY_LISTENERS
          value: http://0.0.0.0:8081

Like this do we have any variables to provide basic authentication? It should ask me for username and password similar to the image.

Thanks for your response!

[OneCricketeer]

SchemaRegistry is asking me username:password

Exactly. So, it's working as expected. The readme shows what files are used and where you find the credentials. Or, the direct file...

https://github.com/confluentinc/confluent-kubernetes-examples/blob/4ceeb1a339c2a690aa121395520e650ae06d8fec/security/plaintext-basic-auth-ConnectAndSchemaRegistry/basicwithroleSR.txt

connect with my existing Kafka cluster?

Yup. Change SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS value to your existing Kafka endpoint

Like this do we have any variables to provide basic authentication?

I don't know what you're referring to. The env?

That's already been shared

authentication.method=BASIC in schema-registry.properties becomes container environment variable

env:
        - name: SCHEMA_REGISTRY_AUTHENTICATION_METHOD
          value: BASIC 

[rudramaniteja]

When i tried the first case, using SchemaRegistry kind of K8s yaml like this

apiVersion: platform.confluent.io/v1beta1
kind: SchemaRegistry
metadata:
  name: schemaregistry
spec:
  replicas: 1
  image:
    application: confluentinc/cp-schema-registry:7.2.0
    init: confluentinc/confluent-init-container:2.4.0
  authentication:
    type: basic
    basic:
      secretRef: basicwithrolesr
      roles:
      - admin
  env:
        - name: SCHEMA_REGISTRY_HOST_NAME
          value: schema-registry
        - name: SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS
          value: "kafka-cluster-kafka-bootstrap.namespace.svc:9092"
        - name: compatibilityLevel
          value: NONE
        - name: SCHEMA_REGISTRY_LISTENERS
          value: http://0.0.0.0:8081

I am getting like this :

And regarding the Envs, Can you provide me how to pass envs for JAAS config file to a pod?

Thanks

[OneCricketeer]

Sorry, I wasn't clear - The environment variables are only meant to be used with the Docker image, standalone, as was your original question. Such as if you used the (now, deprecated) Helm Charts, or created your own deployment definition using the Docker image, not the operator.

This section

spec:
   authentication:
     type: basic

Can only be used with the Operator, and doesn't require a (user supplied) JAAS file, from what I can see in the examples.

By the way, the Operator will require an enterprise license to use beyond the trial period.

I don't know the spec of the SchemaRegistry resource to define custom properties, you'll need to kubectl explain it...

how to pass envs for JAAS config file to a pod?

SCHEMA_REGISTRY_OPTS env is used for any JVM arguments.

You should be able to use a ConfigMap with a volumeMount to mount any files, such as JAAS conf with password file. But I don't think you can template a JAAS config without external tooling like dockerize at runtime of the container (i.e. build your own image). And I'm not sure if the operator supports random mounts into the pod.

Again, Kubernetes related questions aren't "issues" with this repo, in particular. You may find better support on the respective Operator repo or the general Confluent Forum where other support team and community members exist.

[rudramaniteja]

Okay, From this link, Tell me how to pass authentication.realm that will resolve all my problems?

[OneCricketeer]

Is that a property of schema-registry.properties file? If so, then add env-var SCHEMA_REGISTRY_AUTHENTICATION_REALM

[rudramaniteja]

Yes, It's the property of basic authentication.

authentication.method=BASIC
authentication.roles=<user-role1>,<user-role2>,...
authentication.realm=<section-in-jaas_config.conf>

these are the values for basic authentication. When configured with the authentication.method=BASIC, schema registry asks for password prompt. And authentication.roles also I have given. But authentication.realm I'm not getting how to specify Jaas config file as mentioned above.

How to specify Jaas config file? What is the content of it. From the link I posted above, found an example Jaas file.

SchemaRegistry-Props {
  org.eclipse.jetty.jaas.spi.PropertyFileLoginModule required
  file="/path/to/password-file"
  debug="false";
};

In the file path, I have given the absolute path of my password file. And in authentication.realm I have given Jaas config file path.

When tried logging in with the credentials given in the password file, I'm not getting authenticated.

One more case is, from this link, Can you tell me how to specify my existing kafka Endpoint to schema registry(By change if you have Kubernetes schema registry persons)?

[OneCricketeer]

specify my existing kafka Endpoint

apiVersion: platform.confluent.io/v1beta1
kind: SchemaRegistry
metadata:
  name: schemaregistry
  namespace: confluent
spec:
  replicas: 2
  image:
    application: confluentinc/cp-schema-registry:7.2.0
    init: confluentinc/confluent-init-container:2.4.0
  dependencies:
    kafka:
      bootstrapEndpoint: <Your kafka here>

[OneCricketeer]

How to specify Jaas config file?

You shouldn't need to. This section along with the configmap automatically handle setting up the necessary files

spec:
   authentication:
     type: basic

[rudramaniteja]

specify my existing kafka Endpoint

apiVersion: platform.confluent.io/v1beta1
kind: SchemaRegistry
metadata:
  name: schemaregistry
  namespace: confluent
spec:
  replicas: 2
  image:
    application: confluentinc/cp-schema-registry:7.2.0
    init: confluentinc/confluent-init-container:2.4.0
  dependencies:
    kafka:
      bootstrapEndpoint: <Your kafka here>

Is this setup requires a license?

Because, After some days I am getting like below. And pod is not going up

[OneCricketeer]

I already mentioned it required a license after 30 days

#2379 (comment)

[rudramaniteja]

Is there a way to achieve it with open source? Though I am using only Schema registry, using this approach, rest all I have already configured. Please suggest a free way to do.

[OneCricketeer]

There is no open source operator for Confluent. The properties of the Schema Registry should all be the same.

As answered before, you do that by mounting a JAAS file in the container (i.e. Using a configmap or persistent volume) and you add the necessary environment variables to modify the server properties file.

https://stackoverflow.com/questions/58391462/implement-schema-registry-with-a-basic-auth

Or you could run the registry outside of kubernetes first to figure out how you'd configure the same (as above link shows), then you can work on making it work as a container later

[rudramaniteja]

Thanks for the suggestion/help.

Now I am trying this with the server set up. From the doc, I am little bit unclear about authentication.realm=<section-in-jaas_config.conf>

Rest I am giving as:

authentication.method=BASIC
authentication.roles=admin

1. What should I mention as "section in jass config" ?

As in official document JAAS file will be like:

SchemaRegistry-Props {
  org.eclipse.jetty.jaas.spi.PropertyFileLoginModule required
  file="/path/to/password-file"
  debug="false";
};

I have given the same, and password-file, I have given the absolute path of the file.
And It will be like,
barney: changeme,user,developer

2. Is this correct?

export SCHEMA_REGISTRY_OPTS=-Djava.security.auth.login.config=/path/to/the/jaas_config.conf
<path-to-confluent>/bin/schema-registry-start <path-to-confluent>/etc/schema-registry/schema-registry.properties

when should I run this command?

[OneCricketeer]

What should I mention as "section in jass config" ?

Documentation already says. The realm is the part outside of the brackets - SchemaRegistry-Props.

password-file... Is this correct

If you want barney to have password changeme and roles user, developer, then sure. I'd recommend not using plaintext passwords, however.

Alternatively, implement your own replacement class for org.eclipse.jetty.jaas.spi.PropertyFileLoginModule that logs in to some external system rather than use property files and store users, passwords, and roles there.

https://www.eclipse.org/jetty/documentation/jetty-10/operations-guide/index.html#og-jaas-loginmodules

when should I run this command... export

You don't. You add SCHEMA_REGISTRY_OPTS an environment variable to the container.

[rudramaniteja]

since I am using a server, the last one: export environment variables should done right?

[OneCricketeer]

I'm not sure what you mean. Kubernetes is a server. It does not inherit host OS environment variables.

Schema Registry is also server. The documentation is for running Schema Registry directly as a Linux process, not as a container, or within Kubernetes. Those steps need to be translated appropriately, which is to add variables to the container/pod.

[rudramaniteja]

Forget about kubernetes or docker. As you said in previous thread to try the set-up on a server, I'm trying this on a Ubuntu server.

Once I get clarity on server, then I can do the same configuration on Kubernetes.

So on a server, when I run the export command, I'm getting as variables unidentified.

[OneCricketeer]

I see. In that case, a simple test would be

export SCHEMA_REGISTRY_OPTS='-Djava.security.auth.login.config=/path/to/the/jaas_config.conf' 
echo $SCHEMA_REGISTRY_OPTS

Should print back that exported value. Then you may run schema-registry-start

[rudramaniteja]

Steps that I tried from the screenshot:

1. cat of schema registry properties file, you can see the authentication steps that official document says.
2. cat of jaas config file, given the full path of the password file.
3. cat of password file, just for testing I have mentioned the plain.
4. curl login of schema registry with response 401, unauthenticated.
5. echo of schema registry opts, As I have already exported the value.

Hope you are clear with the approach. If I had missed anything, let me know.

Why I still have the problem with login?

Thanks!

[OneCricketeer]

In your OPTS variable, I see you have aut, not auth

[rudramaniteja]

updated the variable, still same.

[OneCricketeer]

I do not believe systemctl inherits local shell variables. Edit your systemd script to add Environment there. Or don't use SystemD and run schema-registry-start directly

[OneCricketeer]

Local files

$ cat registry-*
SchemaRegistry-Props {
  org.eclipse.jetty.jaas.spi.PropertyFileLoginModule required
  file="/tmp/passwords"
  debug="true";
};

barney: changeme,user,developer

Docker Compose

  schema-registry:
    image: confluentinc/cp-schema-registry:7.2.2
    hostname: schema-registry
    container_name: schema-registry
    depends_on:
      - kafka
    ports:
      - "8081:8081"
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: 'kafka:29092'
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081
      SCHEMA_REGISTRY_OPTS: '-Djava.security.auth.login.config=/tmp/registry-jaas.conf'
      SCHEMA_REGISTRY_AUTHENTICATION_REALM:  'SchemaRegistry-Props'
      SCHEMA_REGISTRY_AUTHENTICATION_METHOD: 'BASIC'
      SCHEMA_REGISTRY_AUTHENTICATION_ROLES:  'admin,user,developer'
    volumes:  # refer above
      - $PWD/registry-jaas.conf:/tmp/registry-jaas.conf:ro
      - $PWD/registry-passwords.txt:/tmp/passwords:ro

Start up logs

schema-registry  | ===> Launching ...
schema-registry  | ===> Launching schema-registry ...
schema-registry  | [2022-11-15 01:36:16,022] INFO SchemaRegistryConfig values:
schema-registry  | 	access.control.allow.headers =
schema-registry  | 	access.control.allow.methods =
schema-registry  | 	access.control.allow.origin =
schema-registry  | 	access.control.skip.options = true
schema-registry  | 	authentication.method = BASIC
schema-registry  | 	authentication.realm = SchemaRegistry-Props
schema-registry  | 	authentication.roles = [admin, user, developer]
schema-registry  | 	authentication.skip.paths = []

Inspecting running process (see -Djava.security.auth.login.config)

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
appuser      1  5.6  3.8 3128440 231552 ?      Ssl  01:42   0:07 java -Xmx512M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dschema-registry.log.dir=/usr/bin/../logs -Dlog4j.configuration=file:/etc/schema-registry/log4j.properties -cp :/usr/bin/../package-schema-registry/target/kafka-schema-registry-package-*-development/share/java/schema-registry/*:/usr/bin/../share/java/confluent-security/schema-registry/*:/usr/bin/../share/java/schema-registry-plugins/*:/usr/bin/../share/java/confluent-common/*:/usr/bin/../share/java/confluent-telemetry/*:/usr/bin/../share/java/rest-utils/*:/usr/bin/../share/java/schema-registry/* -Djava.security.auth.login.config=/tmp/registry-jaas.conf io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain /etc/schema-registry/schema-registry.properties

Testing

$ curl localhost:8081/subjects
{"error_code":401,"message":"Unauthorized"}
$ curl -u barney:changeme localhost:8081/subjects
[]

[hafizmujadidKhalid]

@OneCricketeer!
Thanks for sharing all info, I tried all steps and it is not working for me. Here is what i did:

1. setup password file like hafiz: MD5:0192023a7bbd73250516f069df18b500,admin,user
2. setup jaas_conf.conf as

SchemaRegistry-Props {
      org.eclipse.jetty.jaas.spi.PropertyFileLoginModule required
      file="/etc/kafka/secrets/password.txt"
      debug="false";
    }

3. setup environment variables like below

           - name: SCHEMA_REGISTRY_AUTHENTICATION_METHOD
             value: BASIC
           - name: SCHEMA_REGISTRY_AUTHENTICATION_REALM
             value: SchemaRegistry-Props
           - name: SCHEMA_REGISTRY_AUTHENTICATION_ROLES
             value: admin,user
           - name: SCHEMA_REGISTRY_OPTS
             value: "-Djava.security.auth.login.config=/etc/kafka/secrets/jaas_config.conf"

4. Ran schema registry container and checked logs:
   kubectl logs schema-registry-867dbfccff-dcz99 | grep authentication

authentication.method = BASIC
authentication.realm = SchemaRegistry-Props
authentication.roles = [admin, user]
authentication.skip.paths = []
ssl.client.authentication = NONE

5. I checked java perocess and SCHEMA_REGISTRY_OPTS is set correctly
   jps -v

1 SchemaRegistryMain -Xms512M -Xmx2G -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dschema-registry.log.dir=/usr/bin/../logs -Dlog4j.configuration=file:/etc/schema-registry/log4j.properties -Djava.security.auth.login.config=/etc/kafka/secrets/jaas_config.conf
120 Jps -Dapplication.home=/usr/lib/jvm/zulu11-ca -Xms8m -Djdk.module.main=jdk.jcmd

6. I tested it using curl command:
   curl -u "hafiz:admin123" http://schema-registry:8081/subjects
   it gives {"error_code":401,"message":"Unauthorized"}

Do you have any pointer what can be wrong? I set the password plain in password.txt and also md5, nothing worked.
thanks in advance

[OneCricketeer]

I suggest trying to reproduce in the Docker container directly, as I showed.

I don't run the Registry in Kubernetes to be able to help with that.