Proxy only works on ports 80 and 443

[Ryu945]

When I setup Collabora with pini-gh / nginx-proxy and pini-gh / docker-letsencrypt-nginx-proxy-companion . I can get the certificate correctly but the reverse proxy only sets itself up on on port 80 and 443. When I had it publish on another port like 9980, it does not proxy to that port. Here is my file.

version: '3.7' 

services:

  proxy:
    image: pinidh/nginx-proxy:latest
    labels:
    - nextcloud-proxy.nginx-proxy=true
    container_name: nextcloud-proxy
    networks:
      - network1
      - network2
      - network3
    ports:
      - 80:80
      - 443:443
      - 9980:9980
    volumes:
      - nextcloud_dhparam:/etc/nginx/dhparam
      - nextcloud_conf.d:/etc/nginx/conf.d:rw
      - nextcloud_vhost.d:/etc/nginx/vhost.d:rw
      - nextcloud_html:/usr/share/nginx/html:rw
      - nextcloud_certs:/etc/nginx/certs:ro
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/tmp/docker.sock:ro
    restart: unless-stopped

  letsencrypt:
    image: pinidh/letsencrypt-nginx-proxy-companion:latest
    container_name: nextcloud-letsencrypt
    depends_on:
      - proxy
    networks:
      - network1
      - network3
    volumes:
      - nextcloud_acme:/etc/acme.sh
      - nextcloud_certs:/etc/nginx/certs:rw
      - nextcloud_vhost.d:/etc/nginx/vhost.d:rw
      - nextcloud_html:/usr/share/nginx/html:rw
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - NGINX_PROXY_CONTAINER_LABEL=nextcloud-proxy.nginx-proxy
      - NETWORK_ACCESS=internal
    restart: unless-stopped

  db:
    image: mariadb
    container_name: nextcloud-mariadb
    networks:
      - network4
    volumes:
      -  nextcloud_db:/var/lib/mysql
      - /etc/localtime:/etc/localtime:ro
    environment:
      - MYSQL_ROOT_PASSWORD=dragondragon
      - MYSQL_PASSWORD=dradra
      - MYSQL_DATABASE=dragon
      - MYSQL_USER=dragonballz
      - NETWORK_ACCESS=internal
    restart: unless-stopped
  
  app:
    image: nextcloud:latest
    container_name: nextcloud-app
    networks:
      - network2
      - network4
      - network3
    depends_on:
      - letsencrypt
      - proxy
      - db
      - av
      - collabora
    volumes:
      - nextcloud:/var/www/html
      - nextcloud_app_config:/var/www/html/config
      - nextcloud_app_custom_apps:/var/www/html/custom_apps
      - nextcloud_app_data:/var/www/html/data
      - nextcloud_app_themes:/var/www/html/themes
      - /etc/localtime:/etc/localtime:ro
    environment:
      - VIRTUAL_HOST=march.ofthedragon.com
      - LETSENCRYPT_march.ofthedragon.com
      - LETSENCRYPT_DNS_MODE=dns_duckdns
      - LETSENCRYPT_DNS_MODE_SETTINGS=export DuckDNS_Token=atokeneffort
      - LETSENCRYPT_EMAIL=theemailtheemailthethetheemail
      - NETWORK_ACCESS=internal
      - LETSENCRYPT_TEST=true
      - DEBUG=1
    restart: unless-stopped


  av:
    image: mkodockx/docker-clamav:alpine
    container_name: nextcloud-clamav
    networks: 
      - network2
    depends_on:
      - proxy
    environment:
      - NETWORK_ACCESS=internal
    restart: unless-stopped


  collabora:
    image: collabora/code
    container_name: nextcloud-collabora
    networks:
      - network3
    depends_on:
      - letsencrypt
      - proxy
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - VIRTUAL_HOST=data.ballz.com
      - LETSENCRYPT_HOST=data.ballz.com
      - LETSENCRYPT_DNS_MODE=dns_duckdns
      - LETSENCRYPT_DNS_MODE_SETTINGS=export DuckDNS_Token=atokeneffort
      - LETSENCRYPT_EMAIL=theemailtheemailthethetheemail
      - LETSENCRYPT_TEST=true
      - DEBUG=1
      - domain=march\\.ofthedragon\\.com
      - username=datadata
      - password=databallz
      - dictionaries=en de es it fr ja
      - NETWORK_ACCESS=internal
    cap_add:
      - MKNOD
    restart: unless-stopped


volumes:
  nextcloud:
  nextcloud_db:
  nextcloud_conf.d:
  nextcloud_vhost.d:
  nextcloud_html:
  nextcloud_certs:
  nextcloud_app_config:
  nextcloud_app_custom_apps:
  nextcloud_app_data:
  nextcloud_app_themes:
  nextcloud_acme:
  nextcloud_dhparam:
  
  

networks:
  network4:
    internal: true
  network2:
  network1:
  network3:
    internal: true

[pini-gh]

Please see the doc: https://github.com/pini-gh/nginx-proxy#multiple-ports

[Ryu945]

I assume I set that on the Collabora container and not the reverse proxy container?

[pini-gh]

Yes.
You don't need to expose port 9980 on the nginx-proxy side. And you set VIRTUAL_PORT=9980 in the collabora/code environment.

[Ryu945]

If I don't publish port 9980 on the nginx-proxy side then how will I proxy into the container from outside? My understanding is that the browser needs access to Collabora as well as the NextCloud container.

I also tried setting the VIRTUAL_PORT=9980 and it still failed to create a rule in the reverse proxy container that actually used port 9980.

[pini-gh]

The nginx-proxy container should publish ports 80 and 443 only. I don't know about collabora/code, but if it exposes a http service on port 9980, then it should declare VIRTUAL_PORT=9980 in the environment block.
If VIRTUAL_HOST=data.ballz.com, then to access your collabora/code instance you just have to point your browser to http://data.ballz.com/.
Into the nginx-proxy container, check the gererated /etc/nginx/conf.d/default.conf file. It should feature an upstream block like this one:

# data.ballz.com
upstream data.ballz.com {
                                ## Can be connected with "network3" network
                        # nextcloud-collabora
                        server 172.19.0.7:9980;
}

[Ryu945]

It doesn't help. My setup file looks like this. I tried it both ways with the line that says:

                 # nextcloud-collabora
                        server 192.168.32.4 down;

done as

                # nextcloud-collabora
                        server 192.168.32.4:9980 down;

I also tried having this:

        location / {
                proxy_pass https://collabora.ofthedragon.com;
        }

set to

        location / {
                proxy_pass https://collabora.ofthedragon.com:9980;
        }

# If we receive X-Forwarded-Proto, pass it through; otherwise, pass along the
# scheme used to connect to this server
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
  default $http_x_forwarded_proto;
  ''      $scheme;
}
# If we receive X-Forwarded-Port, pass it through; otherwise, pass along the
# server port the client connected to
map $http_x_forwarded_port $proxy_x_forwarded_port {
  default $http_x_forwarded_port;
  ''      $server_port;
}
# If we receive Upgrade, set Connection to "upgrade"; otherwise, delete any
# Connection header that may have been passed to this server
map $http_upgrade $proxy_connection {
  default upgrade;
  '' close;
}
# Apply fix for very long server names
server_names_hash_bucket_size 128;
# Default dhparam
ssl_dhparam /etc/nginx/dhparam/dhparam.pem;
# Set appropriate X-Forwarded-Ssl header
map $scheme $proxy_x_forwarded_ssl {
  default off;
  https on;
}
gzip_types text/plain text/css application/javascript application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
log_format vhost '$host $remote_addr - $remote_user [$time_local] '
                 '"$request" $status $body_bytes_sent '
                 '"$http_referer" "$http_user_agent"';
access_log off;
                ssl_protocols TLSv1.2 TLSv1.3;
                ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';
                ssl_prefer_server_ciphers off;
resolver 127.0.0.11;
# HTTP 1.1 support
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
proxy_set_header X-Forwarded-Ssl $proxy_x_forwarded_ssl;
proxy_set_header X-Forwarded-Port $proxy_x_forwarded_port;
# Mitigate httpoxy attack (see README for details)
proxy_set_header Proxy "";
server {
        server_name _; # This is just an invalid value which will never trigger on a real hostname.
        listen 80 ;
        access_log /var/log/nginx/access.log vhost;
        return 503;
}
# collabora.ofthedragon.com
upstream collabora.ofthedragon.com {
                                # Cannot connect to network of this container
                                server 127.0.0.1 down;
                                # Cannot connect to network of this container
                                server 127.0.0.1 down;
                                ## Can be connected with "shinra_marchingband" network
                # nextcloud-collabora
                        server 192.168.32.4 down;
}
server {
        server_name collabora.ofthedragon.com;
        listen 80  ;
        access_log /var/log/nginx/access.log vhost;
        # Do not HTTPS redirect Let'sEncrypt ACME challenge
        location /.well-known/acme-challenge/ {
                auth_basic off;
                allow all;
                root /usr/share/nginx/html;
                try_files $uri =404;
                break;
        }
        location / {
                return 301 https://$host$request_uri;
        }
}
server {
        server_name collabora.ofthedragon.com;
        listen 443 ssl http2  ;
        access_log /var/log/nginx/access.log vhost;
        # Only allow traffic from internal clients
        include /etc/nginx/network_internal.conf;
        ssl_session_timeout 5m;
        ssl_session_cache shared:SSL:50m;
        ssl_session_tickets off;
        ssl_certificate /etc/nginx/certs/collabora.ofthedragon.com.crt;
        ssl_certificate_key /etc/nginx/certs/collabora.ofthedragon.com.key;
        ssl_dhparam /etc/nginx/certs/collabora.ofthedragon.com.dhparam.pem;
        ssl_stapling on;
        ssl_stapling_verify on;
        ssl_trusted_certificate /etc/nginx/certs/collabora.ofthedragon.com.chain.pem;
        add_header Strict-Transport-Security "max-age=31536000" always;
        location / {
                proxy_pass https://collabora.ofthedragon.com;
        }
}
# march.ofthedragon.com
upstream march.ofthedragon.com {
                                # Cannot connect to network of this container
                                server 127.0.0.1 down;
                                # Cannot connect to network of this container
                                server 127.0.0.1 down;
                                # Cannot connect to network of this container
                                server 127.0.0.1 down;
                                ## Can be connected with "shinra_electric" network
                        # nextcloud-app
                        server 192.168.0.4:80;
                                # Cannot connect to network of this container
                                server 127.0.0.1 down;
                                # Cannot connect to network of this container
                                server 127.0.0.1 down;
                                # Cannot connect to network of this container
                                server 127.0.0.1 down;
                                # Cannot connect to network of this container
                                server 127.0.0.1 down;
                                ## Can be connected with "shinra_marchingband" network
                        # nextcloud-app
                        server 192.168.32.5:80;
}
server {
        server_name march.ofthedragon.com;
        listen 80  ;
        access_log /var/log/nginx/access.log vhost;
        # Do not HTTPS redirect Let'sEncrypt ACME challenge
        location /.well-known/acme-challenge/ {
                auth_basic off;
                allow all;
                root /usr/share/nginx/html;
                try_files $uri =404;
                break;
        }
        location / {
                return 301 https://$host$request_uri;
        }
}
server {
        server_name march.ofthedragon.com;
        listen 443 ssl http2  ;
        access_log /var/log/nginx/access.log vhost;
        # Only allow traffic from internal clients
        include /etc/nginx/network_internal.conf;
        ssl_session_timeout 5m;
        ssl_session_cache shared:SSL:50m;
        ssl_session_tickets off;
        ssl_certificate /etc/nginx/certs/march.ofthedragon.com.crt;
        ssl_certificate_key /etc/nginx/certs/march.ofthedragon.com.key;
        ssl_dhparam /etc/nginx/certs/march.ofthedragon.com.dhparam.pem;
        ssl_stapling on;
        ssl_stapling_verify on;
        ssl_trusted_certificate /etc/nginx/certs/march.ofthedragon.com.chain.pem;
        add_header Strict-Transport-Security "max-age=31536000" always;
        location / {
                proxy_pass http://march.ofthedragon.com;
        }
}

[pini-gh]

             # nextcloud-collabora
                    server 192.168.32.4 down;

The problem is that your nginx-proxy instance can't reach your collabora/code instance. You have to fix it. Have you checked that this container is actually up and running?
It might be easier to debug if you put everything on the same bridge network. You'll be able to experiment moving parts afterwards.

[Ryu945]

Isn't it saying down because it is trying over port 80 and 443 and not 9980?

[pini-gh]

Isn't it saying down because it is trying over port 80 and 443 and not 9980?

Nope. You can check that against the related Go template part:

{{ define "upstream" }}
        {{ if .Address }}
                {{/* If we got the containers from swarm and this container's port is published to host, use host IP:PORT */}}
                {{ if and .Container.Node.ID .Address.HostPort }}
                        # {{ .Container.Node.Name }}/{{ .Container.Name }}
                        server {{ .Container.Node.Address.IP }}:{{ .Address.HostPort }};
                {{/* If there is no swarm node or the port is not published on host, use container's IP:PORT */}}
                {{ else if .Network }}
                        # {{ .Container.Name }}
                        server {{ .Network.IP }}:{{ .Address.Port }};
                {{ end }}
        {{ else if .Network }}
                # {{ .Container.Name }}
                {{ if .Network.IP }}
                        server {{ .Network.IP }} down;
                {{ else }}
                        server 127.0.0.1 down;
                {{ end }}
        {{ end }}

[pini-gh]

I've added a few debug statements to my template to outline what the problem could be. There are 3 cases:

1. The client container exposes only one port: this port will be binded whatever the VIRTUAL_PORT variable holds

# foo.example.com
upstream foo.example.com {
                                ## Can be connected with "reverse-proxy_bridge" network
                                # Exposed ports: [{   80  tcp }]
                                # VIRTUAL_PORT: 9980
                        # foo
                        server 172.19.0.7:80;
}

2. The client container exposes at least two ports and the port number assigned to VIRTUAL_PORT (default=80) is not one of the exposed ports: down

# foo.example.com
upstream foo.example.com {
                                ## Can be connected with "reverse-proxy_bridge" network
                                # Exposed ports: [{   443  tcp } {   80  tcp }]
                                # VIRTUAL_PORT: 9980
                        # foo
                        server 172.19.0.7 down;
}

3. The client container exposes at least two ports and the port number assigned to VIRTUAL_PORT (default=80) is one of the exposed ports: this port will be binded

# foo.example.com
upstream foo.example.com {
                                ## Can be connected with "reverse-proxy_bridge" network
                                # Exposed ports: [{   443  tcp } {   80  tcp } {   9980  tcp }]
                                # VIRTUAL_PORT: 9980
                        # foo
                        server 172.19.0.7:9980;
}

Using docker inspect check which ports are exposed by your collabora/code instance.

[pini-gh]

@Ryu945 any news?

[pini-gh]

@Ryu945, I'll close this ticket in 10 days if it doesn't evolve anymore.

[Ryu945]

I'm going to try your examples to see if I can get 9980 to bind.

I have been looking things up and I think it is related to some custom configuration that Collabora needs but I have not been able to get working.

It is mentioned in this video.

www.youtube.com/watch?v=jBNbOAK-VKQ

[pini-gh]

I've run a simple test with this Docker compose file:

version: "3.5"

services:
  collabora:
    image: collabora/code:latest
    container_name: collabora
    environment:
    - VIRTUAL_HOST=foo.example.com
    - VIRTUAL_PORT=9980
    - LETSENCRYPT_HOST=foo.example.com
    - domain=foo.example.com
    - username=admin
    - password=mysecretpassword
    volumes:
    - config:/etc/loolwsd
    networks:
      reverse_proxy:

volumes:
  config:

networks:
  reverse_proxy:
    external: true
    name: reverse-proxy_bridge

Where reverse-proxy_bridge is the network bridge attached to my nginx-proxy instance. And it just works:

$ curl https://foo.example.com/
OK

$ docker logs collabora 2>&1 | grep 9980
wsd-00006-00006 2020-10-27 19:46:35.520748 [ loolwsd ] INF  Listening to client connections on port 9980| wsd/LOOLWSD.cpp:3866
wsd-00006-00034 2020-10-27 19:46:35.522040 [ prisoner_poll ] INF  Launching forkit process: /usr/bin/loolforkit --losubpath=lo --systemplate=/opt/lool/systemplate --lotemplate=/opt/collaboraoffice6.4 --childroot=/opt/lool/child-roots/ --clientport=9980 --masterport=loolwsd-hxoPTN6d --rlimits=limit_virt_mem_mb:0;limit_stack_mem_kb:8000;limit_file_size_mb:0;limit_num_open_files:0 --version --ui=notebookbar| wsd/LOOLWSD.cpp:1917
Ready to accept connections on port 9980.

Note: I applied these settings into /etc/loolwsd/loolwsd.xml:

ssl.enable=false
ssl.termination=true

as described in possibility number 2 'SSL terminates at the proxy' in this page.

If you have difficulties configuring your collabora/code instance you may find some more valuable help on the Collabora support channels than here.

[Ryu945]

I tried modifying the proxy configuration file located at /etc/nginx/conf.d but when I restart the container, it reverts back to its original setup. How do I modify this containers proxy file?

[pini-gh]

Seriously, do you happen to read documentation? The file /etc/nginx/conf.d/default.conf is generated from /app/nginx.tmpl. So yes, any modification to this file will be overridden.

To set up a custom nginx configuration see https://github.com/pini-gh/nginx-proxy/tree/acme-challenge#custom-nginx-configuration.

[Ryu945]

If that is the case then how is making a volume out of /etc/nginx/conf.d suppose to back up the container when I destroy it and rebuild it again?

[pini-gh]

If that is the case then how is making a volume out of /etc/nginx/conf.d suppose to back up the container when I destroy it and rebuild it again?

I don't understand what your point is.

[Ryu945]

I thought the entire point of creating a volume at /etc/nginx/conf.d was so that when I delete a container, I can use the saved data for building a new container. If this is not the case then what area should I be backing up to save the proxy configuration in a volume?

edit:

I see, I need to add the file proxy.conf so that it replaces default.

[pini-gh]

I'm glad you've been able to find this answer.

Since the topic is not the original one anymore, and I've given you a detailed example answering the question, I'm closing this issue.

[Ryu945]

I figured out one of the problems. Apparently, if you have Collabora on an internal network only then this NGINX container will set the upstream as

upstream collabora.com {
                        server  ip down;
                                # Cannot connect to network of this container
                                server 127.0.0.1 down;
}

When it should be set as:

upstream collabora.com {
                        server  ip;
                                # Cannot connect to network of this container
                                server 127.0.0.1 down;
}

I think this may be an issue with the NGINX container itself. You mentioned earlier that down needed to be removed from that line but now we know why it is being added. Only when collabora is on an internal network and only an internal network does down get added to the line.

I found another issues I am trying to solve. When I put a file in either in /etc/nginx/proxy.conf or /etc/nginx/conf.d/proxy.conf , the container gives complaints about having two files trying to set the same thing. It essentially goes down every setting in the file and complains that it was set elsewhere. I thought simply placing the file in those directories would be enough to tell it to ignore default.conf but apparently not. Any idea how to fix this?

[pini-gh]

I figured out one of the problems. Apparently, if you have Collabora on an internal network only then this NGINX container will set the upstream as

This is nginx-proxy#1132.

[Ryu945]

Do you know why configurations files aren't working like the guide said they should?

edit: I wonder if it could be related to the fact I don't place the proxy.conf file in until after the container has run once.

[pini-gh]

I guess it depends on what you want to define into proxy.conf. I didn't try this feature myself.

I've just published a new release 0.7.0-pini3 which should fix the upstream issue regarding internal networks.