Document how to avoid 502s

[bowei]

From @esseti on September 20, 2017 9:22

Hello,
i've a problem with the ingress and the fact that the 502 page pops up when there are "several" request. I've a JMeter spinning 10 threads for 20 times, and I get more than 50 times the 502 over 2000 calls in total (less than 0,5%).

reading the readme it says
it says that this error is probably due to

The loadbalancer is probably bootstrapping itself.

but the loadbalancer is already there, so does it means that all the pods serving that url are busy? is there a way to avoid the 502 waiting for a pod to be free?

if not, is there a way to personalize the 502 page? because I expose APIs in JSON format, and I would like to show a JSON error rather than a html page.

Copied from original issue: kubernetes/ingress-nginx#1396

[bowei]

From @nicksardo on September 26, 2017 16:12

https://serverfault.com/questions/849230/is-there-a-way-to-use-customized-502-page-for-load-balancer
This is a question for GCP, not the ingress controller. Though, I suggest you investigate why you're getting 502s.

[bowei]

From @esseti on September 29, 2017 9:53

Regarding the several 502, i found out that it's due to how long the LB keeps the connection alive vs what the container provides as keepalive timout. it's explained here (point 3) https://blog.percy.io/tuning-nginx-behind-google-cloud-platform-http-s-load-balancer-305982ddb340

In my case I also added a timout of 5s to the probe, not sure but that solved the 502 (i'm using uwsgi)

[montanaflynn]

I also get google's 502 html error page and would like to understand why and how to avoid it or customize the response. The backend pods have been running without restarting but still maybe 1/1000 requests return a 502. Using GKE with ingress that sends to API pod running nginx.

[esseti]

@montanaflynn have you tried this (point 3) https://blog.percy.io/tuning-nginx-behind-google-cloud-platform-http-s-load-balancer-305982ddb340 ? (i also had to increase the time of probeness, there are my comments on that page).
I've solved the problem with the 502 page, still not able to modified it but that it's not part of kubernetes.

[mikesparr]

Using GKE 1.7.8 (google cloud)

I'm getting these too, and backend services have 2/2 for cluster health and green. Using Ingress with kube-lego and gce for TLS provisioning. One app served by ingress (name-based virtual hosts from single ingress) works fine but other app every other request returning 502 and preventing QA. There were zero issues with either app during initial QA when behind LoadBalancer service. I changed to NodePort services and added Ingress with TLS in front of them and plagued with 502 errors.

Updated liveness and readiness probes, and confirmed 200 responses both at probe URI and / but still getting these errors.

Redacted Ingress Config

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: staging-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: "kubernetes-ingress-stg"
    kubernetes.io/tls-acme: "true"
    kubernetes.io/ingress.class: "gce"
spec:
  tls:
  - hosts:
    - eval.redacted-site1.com
    - eval.redacted-site2.com
    secretName: legacy-tls
  rules:
  - host: eval.redacted-site1.com
    http:
      paths:
      - path: /*
        backend:
          serviceName: site1-app
          servicePort: 80
  - host: eval.redacted-site2.com
    http:
      paths:
      - path: /*
        backend:
          serviceName: site2-app
          servicePort: 80

[montanaflynn]

@esseti I tried increasing the timeout as suggested but still get 502s

Also like @mike-saparov we're using TLS with the ingress (not acme) and NodePort services.

[mikesparr]

I increased keepalive too per recommendation and didn't fix.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[lunemec]

We're also having these issues. Using NodePort on our services, TLS ingress with kube-lego.

We noticed 502 right after this message showed up in kubectl get events:

2m          17d          2637     ingressID                                  Ingress                                                     Normal    Service                 loadbalancer-controller                            default backend set to serviceID

Any ideas how to figure out what is causing these 502's?

[lunemec]

/remove-lifecycle stale