502 Bad Gateway

[rimusz]

I'm getting the bad gateway for Deis apps when using DNS configured access to the platform :

2016/05/06 15:05:14 [error] 25#0: *1706 connect() failed (111: Connection refused) while connecting to upstream, client: 10.244.0.1, server: ~^helloworld\.(?<domain>.+)$, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.247.249.241:80/favicon.ico", host: "helloworld.linxos.net", referrer: "http://helloworld.linxos.net/"
[06/May/2016:15:05:14 +0000] - 10.244.0.1 - - - 502 - "GET /favicon.ico HTTP/1.1" - 328 - "http://helloworld.linxos.net/" - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/601.5.17 (KHTML, like Gecko) Version/9.1 Safari/601.5.17" - "~^helloworld\x5C.(?<domain>.+)$" - 10.247.249.241:80 - helloworld.linxos.net - 0.000 - 0.000
2016/05/06 15:05:16 [error] 25#0: *1704 connect() failed (111: Connection refused) while connecting to upstream, client: 10.240.0.6, server: ~^helloworld\.(?<domain>.+)$, request: "GET / HTTP/1.1", upstream: "http://10.247.249.241:80/", host: "helloworld.linxos.net"

[krancour]

@rimusz that looks like it was caused by the upstream refusing the connection. What happens if you ssh out to some node in the cluster and curl http://10.244.0.1?

[rimusz]

I have the same issue on two GKE clusters

[jchauncey]

Same

[krancour]

Have either of you tried accessing the upstream service directly?

[rimusz]

when the deis-builder.... came in? it was not the case last Saturday for beta3, all demos worked just fine.

[bacongobbler]

The builder domain change does not affect how applications are routed. What @krancour is trying to explain is to manually bypass the router by checking the pod IP address and see if the application itself is routing properly. That would eliminate whether or not the router is the issue here.

[rimusz]

already cleaned up my clusters, will do from the scratch and see what happens

[jchauncey]

Im seeing this issue on apps not deployed through deis but are configured to be routable (grafana for instance)

[krancour]

What @krancour is trying to explain is to manually bypass the router by checking the pod IP address

Or the service IP address. Check both.

[rimusz]

on a fresh kube/deis cluster still the same issue, but curl by pod IP works fine from the node:

$ curl 10.244.2.8
Welcome to Deis!
See the documentation at http://docs.deis.io/ for more information.

[krancour]

@rimusz this is on GKE, right? Can you please document exact steps to reproduce?

[rimusz]

yup, GKE, usual GKE cluster and workflow install steps

[krancour]

What is needed beyond that? Are you using the workflow-dev chart? Is this happening only specific test apps?

[rimusz]

beta 3, latest deis cli, our example apps (example-dockerfile-http, helloworld) Dockefile/docker image get that issue, but the build pack one example-go works fine

[krancour]

Have you tried with the workflow-dev chart?

[rimusz]

not yet

[rimusz]

tried dev chart, controller is crashing there:

$ kd logs deis-controller-26oou
system information:
Django Version: 1.9.5
Python 3.5.1
Django checks:
System check identified no issues (2 silenced).
Health Checks:
Checking if database is alive
There was a problem connecting to the database

deleting the pod did not help

[helgi]

Have you look at if your DB is broken?

[krancour]

I am not able to reproduce this. I've used GKE with the workflow-dev chart, and loaded up the cluster with:

• 10 apps deployed via git push of Dockerfile
• 10 apps deployed via deis pull
• 5 apps built with a buildpack

All 25 apps and Grafana were accessible every time I tried:

for i in $(deis apps | grep "^\w");do curl http://$i.krancour.deis.ninja; done

@jchauncey says he hasn't run into this in about a week now.

@rimusz are you still able to reproduce? If so, I'd want to know how my attempt to reproduce differs from what you are doing.

[rimusz]

@krancour I was able to reproduce it in Berlin yesterday on the last week created cluster from beta3 chart. Bootstrapped a new GKE cluster today and used dev chart and cannot reproduce the problem anymore.
It looks like it was the GKE problem.
I think we can close this one for now.

[krancour]

@rimusz i haven't tried using beta3. It's possible also that if there was a bug, it may have been isolated to that release and fixed (inadvertently) in the meantime. Thanks for following up.