nginx-proxy returns 503 error over http and https #426

leonsp · 2016-04-21T18:23:42Z

I'm setting up nginx-proxy in front of an app server. They're defined in separate compose definitions. For some reason I'm getting a 503, but I don't know why and I've gone over the nginx-proxy docs in detail.

The app server originally served https over 443 with 10443 exposed on the host. I switched to serving http over 80 with 10443 exposed on the host.

I can curl from the app server directly, but curling through nginx-proxy throws up an error

I initially had nginx-proxy on 443, but I switched it to 80 for now.

Until I added default.crt and default.key, I was getting a connection refused error. After adding them, I'm getting a 503.

curl http://foo.example.com:80/apidocs --verbose --insecure
* Hostname was NOT found in DNS cache
*   Trying 10.x.x.x...
* Connected to foo.example.com (10.x.x.x) port 80 (#0)
> GET /apidocs HTTP/1.1
> User-Agent: curl/7.35.0
> Host: foo.example.com
> Accept: */*
>
< HTTP/1.1 503 Service Temporarily Unavailable
* Server nginx/1.9.12 is not blacklisted
< Server: nginx/1.9.12
< Date: Thu, 21 Apr 2016 17:26:16 GMT
< Content-Type: text/html
< Content-Length: 213
< Connection: keep-alive
<
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body bgcolor="white">
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.9.12</center>
</body>
</html>
* Connection #0 to host foo.example.com left intact

Here's my compose definition for nginx-proxy. I'm using network_mode: bridge which is supposed to work even with version: 2.

version: '2'
# Not yet compatible with custom networks in v2 of Compose
services:
  nginx:
    image: jwilder/nginx-proxy
    # Necessary until nginx-proxy fully supports Compose v2 networking
    network_mode: bridge
    ports:
      - "80:80"
    restart: always
    volumes:
      - "certs:/etc/nginx/certs:ro"
      - "nginx-log:/var/log/nginx"
      - "/var/run/docker.sock:/tmp/docker.sock:ro"
volumes:
  certs:
    external: true
  nginx-log:
    external: true

Here's my app server composition:

version: '2'
services:
  database:
    image: sameersbn/postgresql:9.4-13
    restart: always
    # Necessary until nginx-proxy fully supports Compose v2 networking
    network_mode: bridge
    ports:
      - "55433:5432"
    environment:
      - DB_USER=foo
      - DB_PASS=...
      - DB_NAME=foo_staging
      - USERMAP_UID=1000
    volumes:
      - "foo-data:/var/lib/postgresql"

  foo:
    image: private-registry.example.com/dswb/foo:1.4.3
    restart: always
    container_name: "dswb-foo"
    links:
      - "database:database"
    # Necessary until nginx-proxy fully supports Compose v2 networking
    network_mode: bridge
    ports:
      - "10443:80"
    volumes:
      - "certs:/home/rails/webapp/certs"
    environment:
#      - "CERT_NAME=example.com"
      - "VIRTUAL_HOSTNAME=foo.example.com"
      - "VIRTUAL_PORT=80"
      - "VIRTUAL_PROTO=http"
#    command: "bash -c 'rake db:migrate && thin --ssl --ssl-key-file certs/star_example_com.key --ssl-cert-file certs/star_example_com.bundle.crt --port 443 --address 0.0.0.0 start'"
    command: "bash -c 'rake db:migrate && thin --port 80 --address 0.0.0.0 start'"
volumes:
  foo-data:
    driver: local
  certs:
    external: true

The certs are less relevant since I switched to port 80 to debug. I have a wildcard certificate for *.example.com. I made a copy named foo.example.com in case nginx-proxy couldn't find it. I tried both setting and not setting CERT_NAME. I've now also generated the dhparam stuff.

root@8b02a7deb220:/etc/nginx/certs# ls -la
total 48
drwxr-xr-x 2 root root 4096 Apr 21 18:15 .
drwxr-xr-x 4 root root 4096 Apr 21 18:06 ..
-rw------- 1 root root 3575 Apr 21 18:03 example.com.crt
-rw-r--r-- 1 root root  769 Apr 21 18:03 example.com.dhparam.pem
-rw------- 1 root root 1679 Apr 21 18:03 example.com.key
-rw-r--r-- 1 root root 1838 Apr 21 18:03 default.crt
-rw-r--r-- 1 root root 3268 Apr 21 18:03 default.key
-rw------- 1 root root 3575 Apr 21 17:37 foo.example.com.crt
-rw-r--r-- 1 root root  769 Apr 21 18:15 foo.example.com.dhparam.pem
-rw------- 1 root root 1679 Apr 21 17:37 foo.example.com.key

This is the only thing that shows up in the nginx-proxy log when I curl:

nginx.1    | foo.example.com 10.x.x.x - - [21/Apr/2016:17:26:16 +0000] "GET /apidocs HTTP/1.1" 503 213 "-" "curl/7.35.0"

Nothing shows up in app server log, meaning it does not see the request.

How do I debug this? Are there better logs somewhere?

The text was updated successfully, but these errors were encountered:

wader · 2016-04-25T14:27:33Z

Can you paste the generated nginx conf? does it work with version 1 compose config?

leonsp · 2016-04-26T15:02:16Z

Here's nginx.conf:

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    server_names_hash_bucket_size 128;
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}
daemon off;

Where should I be seeing generated conf? There's only this in conf.d:

root@6d8b30a988c2:/etc/nginx/conf.d# ls
default.conf  error_log.conf

default.conf:

# If we receive X-Forwarded-Proto, pass it through; otherwise, pass along the
# scheme used to connect to this server
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
  default $http_x_forwarded_proto;
  ''      $scheme;
}
# If we receive Upgrade, set Connection to "upgrade"; otherwise, delete any
# Connection header that may have been passed to this server
map $http_upgrade $proxy_connection {
  default upgrade;
  '' close;
}
gzip_types text/plain text/css application/javascript application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
log_format vhost '$host $remote_addr - $remote_user [$time_local] '
                 '"$request" $status $body_bytes_sent '
                 '"$http_referer" "$http_user_agent"';
access_log off;
# HTTP 1.1 support
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
server {
    server_name _; # This is just an invalid value which will never trigger on a real hostname.
    listen 80;
    access_log /var/log/nginx/access.log vhost;
    return 503;
}
server {
    server_name _; # This is just an invalid value which will never trigger on a real hostname.
    listen 443 ssl http2;
    access_log /var/log/nginx/access.log vhost;
    return 503;
    ssl_certificate /etc/nginx/certs/default.crt;
    ssl_certificate_key /etc/nginx/certs/default.key;
}

I added error_log.conf while debugging but I'm still not seeing any error messages.

# cat error_log.conf
error_log   /var/log/nginx/error.log   debug;

I suppose there is a weird error in Nginx log output when I look now:

dockergen.1 | 2016/04/26 15:00:11 error inspecting container: 6d8b30a988c2547da9454e7ac417766106949ca423f56e12c20e07c7228c407b: API error (500): devmapper: Unknown device 5c7737cb81b2e86d87112efdce29a7fd4d802e769aeb535efa697e81b05753b6

wader · 2016-04-26T15:12:08Z

default.conf is the generated config and it looks like it does not find any containers to generate config for. To be sure can you try version 1 config? looking at the issue where they talk about the bridge workaround there seems to be some issues with it.

ethiclab · 2016-09-07T16:20:34Z

Hello!

Just to report that:

We have the following docker-compose project and it works fine on:

mac osx: 10.12
docker: Version 1.12.1-beta24 (build: 11525) 0afa5e569ec61f2b8d303e1a722c7e0b0c58d496
docker-compose version 1.8.0, build f3628c7

But we are having a 503 from nginx-proxy on:

ubuntu: Ubuntu 16.04.1 LTS xenial Linux 4.4.0-22-generic #38-Ubuntu SMP Sun Apr 24 20:48:43 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
docker: Docker version 1.12.1, build 23cf638
docker-compose version 1.8.0, build f3628c7

the logs are something like:

forego     | starting nginx.1 on port 5000
forego     | starting dockergen.1 on port 5100
dockergen.1 | 2016/09/07 16:16:56 Generated '/etc/nginx/conf.d/default.conf' from 6 containers
dockergen.1 | 2016/09/07 16:16:56 Running 'nginx -s reload'
dockergen.1 | 2016/09/07 16:16:56 Watching docker events
dockergen.1 | 2016/09/07 16:16:56 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'
dockergen.1 | 2016/09/07 16:16:56 Received event die for container ef50381a5b1a
dockergen.1 | 2016/09/07 16:16:56 Received event start for container 7acd1ebe7c0e
dockergen.1 | 2016/09/07 16:16:56 Received event start for container 34b7a1c6d95a
dockergen.1 | 2016/09/07 16:16:56 Generated '/etc/nginx/conf.d/default.conf' from 7 containers
dockergen.1 | 2016/09/07 16:16:56 Running 'nginx -s reload'
dockergen.1 | 2016/09/07 16:16:57 Received event start for container db6aa098bd70
dockergen.1 | 2016/09/07 16:16:57 Generated '/etc/nginx/conf.d/default.conf' from 8 containers
dockergen.1 | 2016/09/07 16:16:57 Running 'nginx -s reload'
dockergen.1 | 2016/09/07 16:16:57 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'
dockergen.1 | 2016/09/07 16:16:57 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'
nginx.1    | localhost 172.18.0.1 - - [07/Sep/2016:16:17:45 +0000] "GET / HTTP/1.1" 503 213 "-" "Wget/1.17.1 (linux-gnu)"

It could seem that contents of /etc/nginx/conf.d/default.conf is not changing, but looking at the generated file:

    /etc/nginx/conf.d/default.conf

I can see that it has indeed been modified. So it would seem also that for some reason, dockergen is not detecting the modification and thus skipping 'nginx -s reload', but issuing that command by hand does not resolve the problem.

Anyway, I just realized is that the only service that has this problem is nginx-proxy. proxied services seem to work just fine.

Could this be useful for the diagnose?

Cheers,

Edu.

clayrisser · 2016-11-09T21:52:38Z

I am having the same problem.
Ubuntu 16.04
Docker 1.12.3

adamconder · 2016-11-20T11:06:30Z

Same problem here for me:

Running macOS sierra with docker 1.12.13 returning a 503.

starting nginx.1 on port 5000
forego     | starting dockergen.1 on port 5100
dockergen.1 | 2016/11/20 11:02:14 Generated '/etc/nginx/conf.d/default.conf' from 2 containers
dockergen.1 | 2016/11/20 11:02:14 Running 'nginx -s reload'
dockergen.1 | 2016/11/20 11:02:14 Watching docker events
dockergen.1 | 2016/11/20 11:02:14 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'
nginx.1    | localhost 172.17.0.1 - - [20/Nov/2016:11:02:27 +0000] "GET / HTTP/1.1" 503 615 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36"
nginx.1    | localhost 172.17.0.1 - - [20/Nov/2016:11:02:27 +0000] "GET /favicon.ico HTTP/1.1" 503 615 "http://localhost/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36"
nginx.1    | localhost 172.17.0.1 - - [20/Nov/2016:11:02:28 +0000] "GET / HTTP/1.1" 503 615 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36"
nginx.1    | localhost 172.17.0.1 - - [20/Nov/2016:11:02:28 +0000] "GET /favicon.ico HTTP/1.1" 503 615 "http://localhost/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36"
nginx.1    | localhost 172.17.0.1 - - [20/Nov/2016:11:02:29 +0000] "GET / HTTP/1.1" 503 615 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36"
nginx.1    | localhost 172.17.0.1 - - [20/Nov/2016:11:02:29 +0000] "GET /favicon.ico HTTP/1.1" 503 615 "http://localhost/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36"
nginx.1    | localhost 172.17.0.1 - - [20/Nov/2016:11:02:29 +0000] "GET / HTTP/1.1" 503 615 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36"
nginx.1    | localhost 172.17.0.1 - - [20/Nov/2016:11:02:29 +0000] "GET /favicon.ico HTTP/1.1" 503 615 "http://localhost/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36"

jakubgs · 2016-12-12T21:50:13Z

I think I'm having the same issue:

The configuration reloads but nothing changes, the upstream section is still empty:

2016/12/12 21:47:25 Received signal: hangup
2016/12/12 21:47:25 Generated '/etc/nginx/conf.d/default.conf' from 2 containers
2016/12/12 21:47:25 Sending container 'nginx' signal '1'

And I just get 503 from the server.

rottenoats · 2017-01-07T13:54:17Z

+1 Getting the same error using docker nginx-proxy and docker gitlab.

aphillipo · 2017-01-08T09:15:50Z

+1 Also seeing exactly this on Docker for Mac. I'm just going to roll my own I think. Shame this looked very cool awesome!

tpraxl · 2017-02-01T23:48:08Z

I have worked with docker for some time now, but just started working with both docker-compose and jwilder/nginx-proxy today.

I am currently experiencing this issue on Linux Mint (Ubuntu 14.04.1) with docker 1.13.0 and docker-compose 1.9.0 but only when using docker-compose / docker-compose.yml.

When I execute docker run -e VIRTUAL_HOST=working.dev -p 8002:80 --name=working -v $(pwd):/app apache-php-5.4 everything works as expected.

Does this have to do with networking in docker-compose?

I noticed the following when running two containers of interest – one via docker, one via docker-compose:

docker container log (working):

2017/02/01 23:28:20 Received event die for container 014adf4535d3
dockergen.1 | 2017/02/01 23:28:20 Generated '/etc/nginx/conf.d/default.conf' from 1 containers
dockergen.1 | 2017/02/01 23:28:20 Running 'nginx -s reload'

docker-compose container log (not working):

dockergen.1 | 2017/02/01 23:30:25 Received event start for container 8b8d15efedd8
dockergen.1 | 2017/02/01 23:30:25 Generated '/etc/nginx/conf.d/default.conf' from 2 containers
dockergen.1 | 2017/02/01 23:30:25 Running 'nginx -s reload'
dockergen.1 | 2017/02/01 23:30:25 Error running notify command: nginx -s reload, exit status 1

The following log message appeared most of the time in early stages of todays trial/error (not working):

Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload

The error page shows: "503 Service Temporarily Unavailable – nginx/1.11.8"

The tail of /etc/nginx/conf.d/default.conf when both, working and not working, are up:

server {
	server_name _; # This is just an invalid value which will never trigger on a real hostname.
	listen 80;
	access_log /var/log/nginx/access.log vhost;
	return 503;
}
# broken.dev
upstream 5ab555de2c83ec9d6097404c6d21b848d72390a7 {
}
server {
	server_name broken.dev;
	listen 80 ;
	access_log /var/log/nginx/access.log vhost;
	location / {
		proxy_pass http://5ab555de2c83ec9d6097404c6d21b848d72390a7;
	}
}
# working.dev
upstream e4e6bd65aff2b62e7563bbb9f01998d1e9a80314 {
				## Can be connect with "bridge" network
			# working
			server 172.17.0.2:80;
}
server {
	server_name working.dev;
	listen 80 ;
	access_log /var/log/nginx/access.log vhost;
	location / {
		proxy_pass http://e4e6bd65aff2b62e7563bbb9f01998d1e9a80314;
	}
}

I find this line interesting that exists only for the working container: Can be connect with "bridge" network

tpraxl · 2017-02-01T23:57:24Z

Just noticed that there's a workaround for this issue here: #552 (haven't tested it myself yet). However, IMHO it's really just a workaround, because I wouldn't want to refer to externally configured networks in a docker-compose.yml that is shared with other devs who might not use the nginx proxy.

[UPDATE]
The solution to this problem is to add network_mode: "bridge" to the service in question. See some answers here: #502

DivyangGajjar · 2019-10-21T12:06:11Z

503 server error solved:

TRY TO SETUP

https://github.com/presentator/presentator-docker

Solution

If you are configured nginx-proxy server
& and youre doing port forwarding on 8080
and you can see 503 error.
Simply change in nignx-proxy container file
change in /etc/nginx/conf.d/default.conf
Put Your forwarding port my is 8080 on line number 52(ex. Listen 8080)
in (proxy_set_header proxy)
My Problem is solved after this simple changes.
hope this step work for you

diveyez · 2020-01-06T09:36:59Z

503 server error solved:

TRY TO SETUP

https://github.com/presentator/presentator-docker

Solution

1. If you are configured **nginx-proxy** server 
    &  and youre doing port forwarding on 8080 
     and you can see 503 error.

2. Simply change in nignx-proxy container file 
    change in /etc/nginx/conf.d/default.conf 
   Put Your forwarding port my is 8080 on line number 52(**ex. Listen 8080**)
   in (proxy_set_header proxy)

3. My Problem is solved after this simple changes.

4. hope this step work for you

This did not work =(

dnwjn · 2021-02-07T20:10:41Z

503 server error solved:

TRY TO SETUP

https://github.com/presentator/presentator-docker

Solution

If you are configured nginx-proxy server
& and youre doing port forwarding on 8080
and you can see 503 error.

Simply change in nignx-proxy container file
change in /etc/nginx/conf.d/default.conf
Put Your forwarding port my is 8080 on line number 52(ex. Listen 8080)
in (proxy_set_header proxy)

My Problem is solved after this simple changes.

hope this step work for you

I managed to get around this by commenting out the nginx service, exposing the ports on the presentator service directly and pointing my reverse proxy to it.

mjdavies mentioned this issue Mar 15, 2017

503 errors #769

Closed

alesrebec mentioned this issue Mar 28, 2017

503 Service Temporarily Unavailable spleshka/drupal-reactjs-docker#3

Closed

leonsp closed this as completed Oct 21, 2019

jdalrymple mentioned this issue Jul 10, 2020

nginx returns 500 on everything, hard-coded in nginx-config zodern/meteor-up#1030

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nginx-proxy returns 503 error over http and https #426

nginx-proxy returns 503 error over http and https #426

leonsp commented Apr 21, 2016

wader commented Apr 25, 2016

leonsp commented Apr 26, 2016

wader commented Apr 26, 2016

ethiclab commented Sep 7, 2016 •

edited

clayrisser commented Nov 9, 2016

adamconder commented Nov 20, 2016 •

edited

jakubgs commented Dec 12, 2016

rottenoats commented Jan 7, 2017

aphillipo commented Jan 8, 2017

tpraxl commented Feb 1, 2017

tpraxl commented Feb 1, 2017 •

edited

DivyangGajjar commented Oct 21, 2019 •

edited

diveyez commented Jan 6, 2020

503 server error solved:

TRY TO SETUP

Solution

dnwjn commented Feb 7, 2021

503 server error solved:

TRY TO SETUP

Solution

Navigation Menu

nginx-proxy returns 503 error over http and https #426

nginx-proxy returns 503 error over http and https #426

Comments

leonsp commented Apr 21, 2016

wader commented Apr 25, 2016

leonsp commented Apr 26, 2016

wader commented Apr 26, 2016

ethiclab commented Sep 7, 2016 • edited

clayrisser commented Nov 9, 2016

adamconder commented Nov 20, 2016 • edited

jakubgs commented Dec 12, 2016

rottenoats commented Jan 7, 2017

aphillipo commented Jan 8, 2017

tpraxl commented Feb 1, 2017

tpraxl commented Feb 1, 2017 • edited

DivyangGajjar commented Oct 21, 2019 • edited

503 server error solved:

TRY TO SETUP

Solution

diveyez commented Jan 6, 2020

503 server error solved:

TRY TO SETUP

Solution

dnwjn commented Feb 7, 2021

503 server error solved:

TRY TO SETUP

Solution

ethiclab commented Sep 7, 2016 •

edited

adamconder commented Nov 20, 2016 •

edited

tpraxl commented Feb 1, 2017 •

edited

DivyangGajjar commented Oct 21, 2019 •

edited