[nginx] Rate limiting not working

[rochdev]

I tried both ingress.kubernetes.io/limit-connections: 1 and ingress.kubernetes.io/limit-rps: 1 and I am still able to do hundreds of requests per second on the endpoint (tested using curl in parallel).

I can see in the nginx log that it picked up the change (I've added the annotations to an existing resource).

Sample from my ingress resource:

  annotations:
    kubernetes.io/tls-acme: "true"
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/limit-rps: 1

Tested with nginx-ingress-controller versions 0.8.3 and 0.9.0-beta.8 on Kubernetes 1.6.4

[aledbf]

@rochdev please check if the IP addresses in the log are correct (maybe you need to enable proxy protocol if you are running in aws)

[rochdev]

@aledbf The only IP I can see in the logs seems to be an internal IP (10.x.x.x). I am using a TCP load balancer created by Kubernetes in GCP.

[rochdev]

For reference, I started from this example: https://github.com/jetstack/kube-lego/tree/master/examples/nginx

[aledbf]

@rochdev you should use service.beta.kubernetes.io/external-traffic: OnlyLocal in the service to avoid losing the source IP address. https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-typeloadbalancer

[rochdev]

@aledbf From my understanding this requires having an instance of the controller on every node. Is there an alternative without such requirement?

[rochdev]

Also, we only have a single node in development at the moment so according to this document the source IP should not be rewritten in this case.

[rochdev]

After making the change you recommended, I now see the correct source IP in the logs, but the requests are still not limited.

[aledbf]

@rochdev please post the generated nginx.conf from the ingress controller running
kubectl exec <ingress pod> cat /etc/nginx/nginx.conf >ngx.cfg

[rochdev]

@aledbf I cannot really share the entire file, but which part am I looking for exactly?

If it's of any use, there are no limit_req, limit_req_zone or limit_conn anywhere in the file.

[aledbf]

@rochdev without the logs or the generated conf there's not much I can do to help you debug this issue.
Add the flag --v=3 to the descriptor to increase the log level and get the reason why is not reading your annotations. The limit sections are defined here https://github.com/kubernetes/ingress/blob/master/controllers/nginx/rootfs/etc/nginx/template/nginx.tmpl#L363

[rochdev]

With the more verbose log level I was able to find that by setting the annotation to a string instead of a number the rate limiting works properly.

  annotations:
    kubernetes.io/tls-acme: "true"
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/limit-rps: "1"

I am new to Kubernetes. Is this the expected behaviour? Are annotations always strings?

[aledbf]

@rochdev yes. I will add a comment in the docs https://github.com/kubernetes/ingress/blob/master/controllers/nginx/configuration.md#annotations

[aledbf]

Closing. Please reopen is you still have question/issues

[michaelfreund]

I'm having the same issue, that rate limiting is not working.

• Added use-proxy-protocol: "true" to my nginx-configmap
• Added service.beta.kubernetes.io/external-traffic: OnlyLocalto my nginx-service

Here are my config files

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress
  annotations:
    ingress.kubernetes.io/limit-connections: "1"
    ingress.kubernetes.io/limit-rps: "1"
spec:
  rules:
  - host: foo.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx-ingress
          servicePort: 443

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-ingress-configmap
data:
  server-tokens: "false"
  proxy-body-size: 50m
  ssl-dh-param: "default/nginx-dhparam"
  proxy-read-timeout: "3600"
  proxy-send-timeout: "3600"
  use-proxy-protocol: "true"

---
apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress
  labels:
    name: nginx-ingress
  annotations:
    service.beta.kubernetes.io/external-traffic: OnlyLocal
spec:
  type: LoadBalancer
  ports:
    - port: 80
      name: http
    - port: 443
      name: https
  selector:
    app: nginx-ingress

I can confirm that rate limiting rules are written to /etc/nginx/nginx.conf.

Please understand, that I'm not able to share our complete configuration files here. But I will do my best to provide you with any necessary information.

Best
Michael

[rochdev]

@michaelfreund How many concurrent requests did you try in your tests? There is a burst setting of 5 times the configured limit, so in your case it's possible that you would be able to do 5 requests per second for a short amount of time.

The burst setting is unfortunately hardcoded and cannot be configured.

[michaelfreund]

@rochdev I did a ab -kc 100 -n 1000 https://foo.example.com. I'm a little bit confused, as the ingress config requests a new load balancer which appears in GCLB panel, while the nginx service already requested a LB which is working properly.

[rochdev]

@michaelfreund If running in GKE you need to add the kubernetes.io/ingress.class: "nginx" annotation to the Ingress resource, otherwise it falls back on the GCE ingress controller which creates a new load balancer.

See https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/multiple-ingress-controllers

[irvifa]

@rochdev This still isn't working btw, I've tried to add kubernetes.io/ingress.class: "nginx" in my configuration file as well. Any other suggestion?

[rochdev]

@rvu95 Can you post the entire configuration?