CVE-2023-5044: Code injection via nginx.ingress.kubernetes.io/permanent-redirect annotation

[cjcullen]

Issue Details

A security issue was identified in ingress-nginx where the nginx.ingress.kubernetes.io/permanent-redirect annotation on an Ingress object (in the networking.k8s.io or extensions API group) can be used to inject arbitrary commands, and obtain the credentials of the ingress-nginx controller. In the default configuration, that credential has access to all secrets in the cluster.

This issue has been rated High (CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:L/A:L), and assigned CVE-2023-5044.

Affected Components and Configurations

This bug affects ingress-nginx. If you do not have ingress-nginx installed on your cluster, you are not affected. You can check this by running kubectl get po -n ingress-nginx.

If you are running the “chrooted” ingress-nginx controller introduced in v1.2.0 (gcr.io/k8s-staging-ingress-nginx/controller-chroot), command execution is possible but credential extraction is not, so the High severity does not apply.

Multi-tenant environments where non-admin users have permissions to create Ingress objects are most affected by this issue.

Affected Versions

• <v1.9.0

Versions allowing mitigation

• v1.9.0

Mitigation

Ingress Administrators should set the --enable-annotation-validation flag to enforce restrictions on the contents of ingress-nginx annotation fields.

Detection

If you find evidence that this vulnerability has been exploited, please contact security@kubernetes.io

Additional Details

See ingress-nginx Issue #10572 for more details.

Acknowledgements

This vulnerability was reported by Jan-Otto Kröpke (Cloudeteer GmbH)

Thank You,
CJ Cullen on behalf of the Kubernetes Security Response Committee

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ealasgarov]

@cjcullen did you mean version 1.19, not 1.9, right?

[joshsleeper]

@ealasgarov I think 1.9.0 is correct and intended, as it's referring to the controller version not the underlying nginx version.

I'm assuming that since 1.9.0 of the controller is when the --enable-annotation-validation flag + feature was added

https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.9.0

[havenotfear]

Is this only affecting deployments where the nginx.ingress.kubernetes.io/permanent-redirect has been added to the ingress annotations?

[stromvirvel]

I'd like to set --enable-annotation-validation, but its impact on the existing Ingress objects is unclear to me. I haven't found any documentation on that feature, except this single sentence of the CLI argument.
May somebody be so kind and give us further context?

[cilindrox]

@stromvirvel each annotation has a risk weight/grade - the annotation validation flag alone won't be of much use, since the default threshold is Critical, so you need to enable both --enable-annotation-validation and add the threshold for risk-level under the configmap, eg annotations-risk-level: High.

As for figuring out the threshold, you'll need to go through the codebase, afaik there's no documentation for each of these - best I could find was this test file: kubernetes/ingress-nginx@c5f348e#diff-1cf51e128ca991f6d8ea012512e57cf68e224e64ff68e2401d9a598e4da98837R70-R74

There you can see nginx.ingress.kubernetes.io/auth-signin (authReqSigninAnnotation) has a Risk of High (AnnotationRiskHigh), so you'd configure annotations-risk-level as High (ie: the highest risk among all of your used annotations). Hope this is somewhat clear

[magustin-telus]

I'd like to set --enable-annotation-validation, but its impact on the existing Ingress objects is unclear to me. I haven't found any documentation on that feature, except this single sentence of the CLI argument. May somebody be so kind and give us further context?

it is in the values.yaml.

[prashanthkasamsetty]

If anyone is not sure on how to enable that annotation or where this annotation will be set, check info mentioned below.

This annotation is set as arg in the container level of the nginx

this annotation is not set by default and needs to be enabled explicitly.
Highlighted below is the value for the --enable-annotation-validation parameter that should be passed with true while you are installing.

Through terraform, you need to put below in your '.tf' file
set {
name = "controller.enableAnnotationValidations"
value = true
}

[rikatz]

As for figuring out the threshold, you'll need to go through the codebase, afaik there's no documentation for each of these - best I could find was this test file: kubernetes/ingress-nginx@c5f348e#diff-1cf51e128ca991f6d8ea012512e57cf68e224e64ff68e2401d9a598e4da98837R70-R74

@cilindrox you are right! I have plans to automate the documentation generation for this annotations (risk, description and eventually the validation) but didn't had the opportunity to do it yet (tho I've left the majority of code ready for it).

Sorry for that, we had to rush on implementation and missed this

[rikatz]

/assign @cpanato @rikatz @strongjz @tao12345666333

[rikatz]

Hey folks,

As this CVE has been opened for 1 week now, I'm closing the issue.

The description of the issue contains all the required mitigations, and we plan in future releases to turn the validation on by default and also implement more safety measures.

Thank you all for using the project, and for your continuous support for us.

/close

[k8s-ci-robot]

@rikatz: Closing this issue.

In response to this:

Hey folks,

As this CVE has been opened for 1 week now, I'm closing the issue.

The description of the issue contains all the required mitigations, and we plan in future releases to turn the validation on by default and also implement more safety measures.

Thank you all for using the project, and for your continuous support for us.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[nguyenthai0107]

Hello @prashanthkasamsetty @rikatz @longwuyuan
depend on CVE-2021-25742 ( This issue has been rated High )
but when I edited annotations-risk-level: High and set --enable-annotation-validation=true , look like this wrong and cannot create ingress object. Im not sure CVE-2021-25742 was rated correctly.
Here are details

1. Configmap Ingress Nginx Controller

apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  allow-snippet-annotations: "true"
  annotations-risk-level: High

2. Deployment Ingress Nginx Controller

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/instance: platform
      app.kubernetes.io/component: controller
  replicas: 5
  revisionHistoryLimit: 10
  minReadySeconds: 0
  template:
    spec:
      containers:
      - name: controller
        image: registry.k8s.io-ingress-nginx-controller:v1.9.5
        imagePullPolicy: IfNotPresent
        lifecycle:
          preStop:
            exec:
              command:
              - /wait-shutdown
        args:
        - /nginx-ingress-controller
        - **--enable-annotation-validation=true**
        - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
        - --election-id=ingress-nginx-leader
        - --controller-class=k8s.io/ingress-nginx
        - --ingress-class=nginx
        - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
        - --validating-webhook=:8443
        - --validating-webhook-certificate=/usr/local/certificates/cert
        - --validating-webhook-key=/usr/local/certificates/key
        - --enable-ssl-passthrough=true

3. Deploy app with Ingress object which got issue.
   Exactly app is deploy Kiali inside namespace Istio

   ingress:
      class_name: "nginx"
      enabled: true
      # default: override_yaml is undefined
      override_yaml:
        metadata:
          annotations:
              nginx.ingress.kubernetes.io/configuration-snippet: |
              proxy_set_header "X-B3-Sampled" "1";
        spec:
          rules:
          - host: kiali.example.com
            http:
              paths:
              - backend:
                  service:
                    name: kiali
                    port:
                      number: 20001
                path: /
                pathType: Prefix
          tls:
          - hosts:
            - kiali.example.com
            secretName: "kiali-tls"`

• Components Used:
  Nginx Ingress Controller 1.9.5
  Helm version 4.9.0
  Kiali Operator v1.77.0

`/nginx-ingress-controller 
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.9.5
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

Here is the issue which use command line kubectl describe kiali -n istio and ingress of Kiali wasn't created.

Status:
  Conditions:
    Message:               
    Reason:                
    Status:                False
    Type:                  Successful
    Message:               Running reconciliation
    Reason:                Running
    Status:                False
    Type:                  Running
    Ansible Result:
      Changed:             5
      Failures:            1
      Ok:                  64
      Skipped:             51
    Message:               Failed to patch object: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"admission webhook \\"validate.nginx.ingress.kubernetes.io\\" denied the request: annotation group ConfigurationSnippet contains risky annotation based on ingress configuration","reason":"BadRequest","code":400}\n'
    Reason:                Failed
    Status:                True
    Type:                  Failure
  Deployment:
    Instance Name:  kiali
    Namespace:      istio
  Environment:
    Is Kubernetes:       true
    Kubernetes Version:  1.24.17
    Operator Version:    v1.77.0
  Progress:
    Duration:    0:00:26
    Message:     5. Creating core resources
  Spec Version:  default
Events:          <none>

But when set to Critical or delete annotations-risk-level in configmap, the issue was gone and working fine.
Just got stuck when set to High, Medium or Low.
So please kindly for take a look.
Thank you guys.

[cji]

/transfer kubernetes

[cji]

/area security
/kind bug
/committee security-response
/triage accepted
/lifecycle frozen
/label official-cve-feed