Deis app responds to health check but router returns 503 Service Temporarily Unavailable

[olalonde]

I just successfully deployed a new release with deis pull ... but when accessing the app from my browser, I am getting a 503 Service Temporarily Unavailable nginx error page.

The app logs show that it is responding to health check requests:

$ deis logs
myapp-cmd-2847749395-jx1u0 -- ::ffff:10.244.1.1 - - [24/Oct/2016:11:36:19 +0000] "GET /_health HTTP/1.1" 200 - "-" "Go 1.1 package http"
myapp-cmd-2847749395-jx1u0 -- ::ffff:10.244.1.1 - - [24/Oct/2016:11:36:29 +0000] "GET /_health HTTP/1.1" 200 - "-" "Go 1.1 package http"
myapp-cmd-2847749395-jx1u0 -- ::ffff:10.244.1.1 - - [24/Oct/2016:11:36:39 +0000] "GET /_health HTTP/1.1" 200 - "-" "Go 1.1 package http"
myapp-cmd-2847749395-jx1u0 -- ::ffff:10.244.1.1 - - [24/Oct/2016:11:36:49 +0000] "GET /_health HTTP/1.1" 200 - "-" "Go 1.1 package http"

$ deis config | grep PORT
PORT                        5000

Some logs from deis router:

[2016-10-24T11:38:24+00:00] - myapp-ui - 172.20.0.83 - - - 503 - "GET / HTTP/1.1" - 792 - "http://myapp.com/" - "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36" - "myapp.com" - -- myapp.com - - - 0.000
[2016-10-24T11:38:29+00:00] - myapp-ui - 172.20.0.131 - - - 503 - "GET / HTTP/1.1" - 390 - "-" - "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)" - "myapp.com" - - - myapp.com - - - 0.000
[2016-10-24T11:38:52+00:00] - deis/deis-controller - 172.20.0.147 - - - 201 - "POST /v2/apps/myapp-ui/config/ HTTP/1.1" -2298 - "-" - "Deis Client v2.7.0" - "~^deis\x5C.(?<domain>.+)$" - 10.0.34.240:80 - deis.d.myapp.com - 82.117 - 82.183
[2016-10-24T11:38:52+00:00] - deis/deis-controller - 172.20.0.131 - - - 200 - "GET /v2/apps/myapp-ui/config/ HTTP/1.1" - 1820 - "-" - "Deis Client v2.7.0" - "~^deis\x5C.(?<domain>.+)$" - 10.0.34.240:80 - deis.d.myapp.com - 0.031 - 0.031
[2016-10-24T11:38:54+00:00] - deis/deis-controller - 10.244.0.1 - - - 200 - "GET /v2/apps/myapp-ui/config/ HTTP/1.1" - 1820 - "-" - "Deis Client v2.7.0" - "~^deis\x5C.(?<domain>.+)$" - 10.0.34.240:80 - deis.d.myapp.com - 0.019 - 0.019
[2016-10-24T11:39:29+00:00] - myapp-ui - 172.20.0.147 - - - 503 - "GET / HTTP/1.1" - 390 - "-" - "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)" - "myapp.com" - - - myapp.com - - - 0.000
[2016-10-24T11:39:41+00:00] - myapp-ui - 172.20.0.83 - - - 503 - "GET /c/dJMPJb HTTP/1.1" - 390 - "-" - "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" - "myapp.com" - - - myapp.com - - - 0.000

But it doesn't show any of the request coming from my browser. So I'm guessing the deis router is having trouble with something. I tried to delete the deis-router pod but still same problem. Any idea?

[olalonde]

Fixed by adding a Procfile with web process and deis ps:scale cmd=0 web=1

[olalonde]

@bacongobbler is this a new requirement? I thought it was fine to use cmd as the web process.

[bacongobbler]

Nope! It should be fine with cmd as the web process. I just closed it as I assumed it was an app issue with booting the process. Please re-open if you have more logs to share about how it failed to return a 200 OK with the cmd process

[kmala]

do you have a procfile with web type during the deis pull?

[olalonde]

I didn't have a Procfile when it failed. But it was the first deis pull for that app, previously I did git push deploys, maybe that's the reason? Maybe this should be tested in e2e tests (git push deploy followed by deis pull deploy)? I don't think it was a booting error because I could see the successful health check requests in my app logs. Anyways, I'm going to comment here if I have this issue again.

[davidchua]

I am having this issue as well.

I've tried with both having a Procfile with a cmd calling the same CMD in the dockerfile and also without a Procfile.

Application is running. Like @olalonde, previously I was git pushing from a buildpack before switching to a Dockerfile method.

[jdotjdot]

I'm also having this issue. Is there any resolution?

[Cryptophobia]

This is issue is fucking insane. 🔫 💀 :

[Cryptophobia]

I was able to reproduce this after many hours. Please reopen for the record!

When a push with a Procfile/Heroku buildpack and then with Dockerfile happen on the same application, the service can reach a state 503 Service Unavailable. I had to recreate the app with a new name and then push there only with the relevant Dockerfile.

Seems like Procfile/Heroku Buildpack pushes can make the application have a dirty state.

The whole time I was trying to deploy for hours and thinking this was host-header problem with my application and the server hosting it inside the docker container. But it turns out the 503 Service Temporarily Unavailable is a problem with Deis.

[jdotjdot]

I would second reopening the issue. We literally have had to avoid using Deis because of it, and instead are using vanilla Kubernetes.

[bacongobbler]

This should've been fixed with deis/controller#1201 and we closed this because it was a duplicate of #658. I'd take a look at that PR and see if there's some edge case missed there.

The code in question would be here: https://github.com/mboersma/controller/blob/master/rootfs/api/models/app.py#L478-L492

[Cryptophobia]

This issue was moved to teamhephy/workflow#1