JWT plugin failing to fetch existing secret

[CristianPupazan]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

I am trying to configure the JWT plugin following the instructions here.

I am having trouble provisioning the JWT credentials.

I have tried creating the secret in various ways, all with the same outcome:

• using the kubectl create secret --from-literal as per the instructions
• or by encoding the values and creating a secret.yaml as described here.
• creating it via Terraform and reading the rsa_public_key it from a file

I see the following errors:

➜  ~ kubectl get events -A --field-selector='reason=KongConfigurationTranslationFailed'
1s          Warning   KongConfigurationTranslationFailed   kongconsumer/quiz-architect   credential "qar-jwt" failure: failed to fetch secret: Secret default/qar-jwt not found

Log from the ingress controller:

│ kong-internal-stg-kong-6c95dfddfb-z58sk ingress-controller time="2023-09-18T10:15:53Z" level=error msg="resource processing failed: │
│  credential \"qar-jwt\" failure: failed to fetch secret: Secret default/qar-jwt not found" GVK="configuration.konghq.com/v1, Kind=K │
│ ongConsumer" name=quiz-architect namespace=default

The proxy gives the following error when providing the jwt token:

{
"message": "No credentials found for given 'iss'"
}

Note that the secret is there:

➜  ~ kubectl get secret qar-jwt
NAME      TYPE     DATA   AGE
qar-jwt   Opaque   4      20m

➜  ~ kubectl get secret qar-jwt -o jsonpath='{.data}'
{"algorithm":"UlMyNTY=","key":"BASE_64_ENCODED_KEY","kongCredType":"and0","rsa_public_key":"BASE_64_ENCODEDPUB_KEY"}%

The consumer looks like this:

➜  ~ kubectl describe kongconsumer quiz-architect
Name:         quiz-architect
Namespace:    default
Labels:       <none>
Annotations:  kubernetes.io/ingress.class: kong
API Version:  configuration.konghq.com/v1
Credentials:
  qar-jwt
custom_id:  quiz-architect
Kind:       KongConsumer
Metadata:
  Creation Timestamp:  2023-09-15T13:47:20Z
  Generation:          2
  Managed Fields:
    API Version:  configuration.konghq.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:custom_id:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
          f:kubernetes.io/ingress.class:
      f:username:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2023-09-15T13:47:20Z
    API Version:  configuration.konghq.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:credentials:
    Manager:         kubectl-patch
    Operation:       Update
    Time:            2023-09-15T14:02:06Z
  Resource Version:  113658890
  UID:               5b0413db-7e8c-4e5e-822e-c6b292285937
Username:            quiz-architect
Events:
  Type     Reason                              Age                      From         Message
  ----     ------                              ----                     ----         -------
  Warning  KongConfigurationTranslationFailed  21m (x81002 over 2d19h)  kong-client  credential "qar-jwt" failure: failed to fetch secret: Secret default/qar-jwt not found
  Warning  KongConfigurationTranslationFailed  21m (x81002 over 2d19h)  kong-client  credential "qar-jwt" failure: failed to fetch secret: Secret default/qar-jwt not found
  Warning  KongConfigurationTranslationFailed  78s (x402 over 21m)      kong-client  credential "qar-jwt" failure: failed to fetch secret: Secret default/qar-jwt not found
  Warning  KongConfigurationTranslationFailed  72s (x402 over 21m)      kong-client  credential "qar-jwt" failure: failed to fetch secret: Secret default/qar-jwt not found

Expected Behavior

KONG should be able to fetch the secret and validate the JWT token.

Steps To Reproduce

1. Configure the Kong plugin:


apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: jwt-auth-plugin
  annotations:
    kubernetes.io/ingress.class: "kong"
plugin: jwt
config:
    claims_to_verify: ["exp"]

2. Associate the plugin with an existing service created via ingress rules:

kubectl annotate service your_service konghq.com/plugins=jwt-auth-plugin

3. Provision a consumer

---
apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: your_consumer_name
  annotations:
    kubernetes.io/ingress.class: kong
username: your_consumer_name
custom_id: your_consumer_name

4. Provision JWT credentials

apiVersion: v1
kind: Secret
metadata:
  name: consumer-jwt
  annotations:
    kubernetes.io/ingress.class: "kong"
type: Opaque
data:
  kongCredType: and0
  key: BASE_64_ENCODED_KEY
  algorithm: UlMyNTY=
  rsa_public_key: |   
BASE_64_ENCODED_PUB_KEY

5. Assign the credentials

 kubectl patch --type json kongconsumer your_consumer_name \                           
  -p='[{
    "op":"add",
    "path":"/credentials",
    "value":consumer-jwt
  }]'

### Kong Ingress Controller version

```shell
app.kubernetes.io/name=kong                                                                                 
app.kubernetes.io/version=3.3                                                                               
helm.sh/chart=kong-2.26.5

Image: kong/kubernetes-ingress-controller:2.11

Kubernetes version

kubectl version: 

Client Version: v1.25.9
Server Version: v1.24.16-eks-2d98532

### Anything else?

_No response_

[programmer04]

Hello, thanks for the report @CristianPupazan. In Steps To Reproduce I can see that for KongConsumer you don't specify the field namespace, but for Secret the namespace kube-system is specified. They are expected to be in the same namespace. Please let me know if it resolves an issue

[CristianPupazan]

Hi Jakub,
I made a mistake there copying the wrong instructions. I've updated the Steps to reproduce and sorry for the confusion. In reality both the KongConsumer and the secret are in the same namespace:

ingress-controller time="2023-09-22T11:35:29Z" level=error msg="resource processing failed: credential \"qar-jwt\" failure: failed to fetch secret: Secret default/qar-jwt not found" GVK="configuration.konghq.com/v1, Kind=KongConsumer" name=quiz-architect namespace=default

Any idea what else it could be I am missing?

[programmer04]

Hello Cristian,
I tried to follow the steps provided by you and it works for me. Could you provide an end-to-end example? I mean that I only copy the whole content without any adjustments/changes and apply it to K8s with KIC installed and I will see the misconfiguration. You can also share the helm install command used for KIC installation with all provided parameters

[CristianPupazan]

Hi Jakub,
See below the end-to-end example and the KIC installation with the parameters used. I re-installed the KIC just now, and went through the end-to-end example provided and was able to replicate the problem.

Install ingress controller:

helm repo add kong https://charts.konghq.com
helm repo update
helm install kong-internal -f values.yaml kong/kong

where values.yaml:

image:
  repository: kong
  tag: "3.3"

# Kong pod count.
# It has no effect when autoscaling.enabled is set to true
replicaCount: 2

env:
  prefix: /kong_prefix/
  database: "off"

ingressController:
  enabled: true
  # https://github.com/Kong/charts/commit/d926656bbca89430292e3d882ec0c56edbbbfa3a
  installCRDs: false

admin:
  tls:
    parameters: []

proxy:
  # Enable creating a Kubernetes service for the proxy
  enabled: true
  type: LoadBalancer
  # To specify annotations or labels for the proxy service, add them to the respective
  # "annotations" or "labels" dictionaries below.
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: true
  labels:
    enable-metrics: "true"
  stream:
    # Set the container (internal) and service (external) ports for this listen.
    # These values should normally be the same. If your environment requires they
    # differ, note that Kong will match routes based on the containerPort only. 5672
    - containerPort: 5672
      servicePort: 5672
      protocol: TCP
      # Optionally set a static nodePort if the service type is NodePort
      # nodePort: 32080
      # Additional listen parameters, e.g. "ssl", "reuseport", "backlog=16384"
      # "ssl" is required for SNI-based routes. It is not supported on versions <2.0
      parameters: [ ]

End to end config:

kubectl apply -f kong-configuration.yaml

where kong-configuration.yaml:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-app
  template:
    metadata:
      labels:
        app: test-app
    spec:
      containers:
        - name: test-app-container
          image: nginx:latest
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: test-app-service
spec:
  selector:
    app: test-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
      nodePort: 30111
  type: NodePort
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: kong
  name: test-ingress
  namespace: default
spec:
  rules:
    - host: test.mydomain.co.uk
      http:
        paths:
          - backend:
              service:
                name: test-app-service
                port:
                  number: 80
            pathType: Prefix
            path: /
---
apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: test-consumer
  annotations:
    kubernetes.io/ingress.class: kong
username: test-consumer
custom_id: test-consumer
---
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: jwt-auth-plugin
  annotations:
    kubernetes.io/ingress.class: "kong"
plugin: jwt
config:
    claims_to_verify: ["exp"]
---
apiVersion: v1
kind: Secret
metadata:
  name: consumer-jwt
  annotations:
    kubernetes.io/ingress.class: "kong"
type: Opaque
data:
  kongCredType: and0
  key: aHR0cHM6Ly9jb2duaXRvLWlkcC5ldS13ZXN0LTEuYW1hem9uYXdzLmNvbS9ldS13ZXN0LTFfMXJoczdOTHZ2
  algorithm: UlMyNTY=
  rsa_public_key: |
    LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0KTUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF5OVV5RHk5c0lnN3JUSjlncXc1NgpXSXlaVDdFdytybWpjMHgvUVJ4cm1aMnF2YU9sK05NaXVrRjI4UUZUM2QxNXhBd3NtQ1Q0QVp0dFk3dHZhV2ZBCm91aWc0b0cyMStFRXBkc2JnY21YcnB2SEVMcFU3V0dISXBCckZMekl1d0NFUDJXV1ZobWRxN01QTG5tdmd3U2IKMUZocTFwMWhWb0NDZ1VlcFRyZWZCSEh4elo3cXJURENndjhQK1Z5Ty9oUCt1WkJJK0ZoamY4bDQ4VEx2SzJnYwpGMGF6YUY3NHpuYU9hOVFGQ2JoMStITkdwRTdWblBlZEw2Q01tUTJ1MzRRUUx6U2JybFdZRzhYcFRkK1RRWTZtCnZXbFJvc0F6Z3l3Lzhzd1p5bzl0Q1g1YURXSHVQRlBuNEp3VU5WUVVyMWJnTDE4V3cvRjZFVWJXVlVvbUxpZlMKUndJREFRQUIKLS0tLS1FTkQgUFVCTElDIEtFWS0tLS0t

Annotate the service

kubectl annotate service test-app-service konghq.com/plugins=jwt-auth-plugin

Add credentials to the consumer

kubectl patch --type json kongconsumer test-consumer \
  -p='[{
    "op":"add",
    "path":"/credentials",
    "value":["consumer-jwt"]
  }]'

Check logs:

ingress-controller time="2023-09-25T15:20:09Z" level=error msg="resource processing failed: credential \"consumer-jwt\" failure: failed to fetch s ││ ecret: Secret default/consumer-jwt not found" GVK="configuration.konghq.com/v1, Kind=KongConsumer" name=test-consumer namespace=default

[CristianPupazan]

Also might be worth mentioning that I had to set ingressController.installCRDs to false as per instructions here.

[rainest]

Are your CRDs up to date? We had a similar report internally that cleared after getting the latest CRDs, though it's unclear why (we don't know of a reason why CRD changes would prevent the controller from finding a Secret).

the installCRDs setting isn't actually related to this--it's related to very old installations and hasn't been required for new ones in some time. Helm 3 provides no CRD management after initial install, and unfortunately no official mechanism to warn users when there are updates available.

What do you get for this before upgrading?

$ kubectl get crd kongconsumers.configuration.konghq.com -ojson | grep '"controller-gen.kubebuilder.io/version":' 
            "controller-gen.kubebuilder.io/version": "v0.13.0",

That may let us roughly pinpoint which version of the CRDs you have to try and understand how it's breaking better.

[CristianPupazan]

It was the CRDs version indeed. I was running on some old version. I just assumed these got updated when we upgraded the ingress via helm.

Before:

➜  ~ kubectl get crd kongconsumers.configuration.konghq.com -ojson | grep '"controller-gen.kubebuilder.io/version":'
            "controller-gen.kubebuilder.io/version": "v0.9.2"

I run the following to upgrade the CRDs:

➜  ~ kubectl apply -f https://raw.githubusercontent.com/Kong/charts/main/charts/kong/crds/custom-resource-definitions.yaml

Current version:

            "controller-gen.kubebuilder.io/version": "v0.13.0",

I've tested and the JWT gets validated now. Thank you all for your help!

[AlexZhenWang]

Hi, I got this error again. The CRDs version is v0.13.0 on my side.