502 error after watchtower pulls image changes (NGINX, DOCKER)

[WKampel]

My NGINX/Docker based server throws a 502 error after watchtower pulls image changes. The running front/back end containers get updated and are ran (I can navigate manually to the ports they are run on to verify), but NGINX throws a 502 instead of serving the frontend.

I'm forced to restart docker-compose after the image pulls in order to fix this issue.

I have tried adding resolver 127.0.0.11 valid=10s; to my NGINX config file, but this did not solve the issue. I'm assuming the issue is some sort of NGINX cache, but I'm not sure.

Steps to reproduce

Setup Watchtower, NGINX and node container in docker-compose on server. Make image changes to DockerHub and see NGINX serve 502 error.

Expected behavior

I expect NGINX to server the updated containers.

Screenshots

No response

Environment

Ubuntu 18.04
Docker 20.10.18

Your logs

time="2022-10-18T00:23:28Z" level=info msg="Session done" Failed=0 Scanned=5 Updated=0 notify=no
time="2022-10-18T00:23:57Z" level=info msg="Session done" Failed=0 Scanned=5 Updated=0 notify=no
time="2022-10-18T00:24:27Z" level=info msg="Session done" Failed=0 Scanned=5 Updated=0 notify=no
time="2022-10-18T00:24:58Z" level=info msg="Session done" Failed=0 Scanned=5 Updated=0 notify=no
time="2022-10-18T00:25:27Z" level=info msg="Session done" Failed=0 Scanned=5 Updated=0 notify=no
time="2022-10-18T00:25:57Z" level=info msg="Session done" Failed=0 Scanned=5 Updated=0 notify=no
time="2022-10-18T00:26:38Z" level=info msg="Found new wkampel/spartan-backend:latest image (d8a14f9ace05)"
time="2022-10-18T00:26:47Z" level=info msg="Found new wkampel/spartan-frontend:latest image (cc3b5db27f8a)"
time="2022-10-18T00:26:48Z" level=info msg="Stopping /devops_frontend_1 (edc4f4528b40) with SIGTERM"
time="2022-10-18T00:26:49Z" level=info msg="Stopping /devops_backend_1 (13742b99736b) with SIGTERM"
time="2022-10-18T00:26:50Z" level=info msg="Creating /devops_backend_1"
time="2022-10-18T00:26:51Z" level=info msg="Creating /devops_frontend_1"
time="2022-10-18T00:26:54Z" level=info msg="Removing image 953ae2975007"
time="2022-10-18T00:26:55Z" level=info msg="Removing image 9a22fac4b128"
time="2022-10-18T00:26:55Z" level=info msg="Session done" Failed=0 Scanned=5 Updated=2 notify=no
time="2022-10-18T00:26:57Z" level=info msg="Session done" Failed=0 Scanned=5 Updated=0 notify=no
time="2022-10-18T00:27:27Z" level=info msg="Session done" Failed=0 Scanned=5 Updated=0 notify=no

Additional context

nginx.conf

upstream frontend {
  server frontend:3000;
}

upstream backend {
  server backend:3001;
}

server {
  listen 80;
  listen 443 ssl; 

  ssl_certificate /etc/letsencrypt/live/myurl.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/myurl.com/privkey.pem;

  resolver 127.0.0.11 valid=10s;

  server_name myurl.com www.myurl.com;

  location ^~ /.well-known {
      allow all;
      root  /data/letsencrypt/;
  }

  location / {
      proxy_pass http://frontend;
  }

  location /sockjs-node {
      proxy_pass http://frontend;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "Upgrade";
  }
  
  location /backend {
      rewrite /backend/(.*) /$1 break;
      proxy_pass http://backend;
  }
}

[piksel]

The easiest way to solve this would be to add a depends-on label to the frontend/backend containers:

  labels:
    com.centurylinklabs.watchtower.depends-on: nginx

That would recreate the nginx container as well when there are updates on backend or frontend.
The alternative is to dig around in how nginx resolves names and docker internals 🤷‍♀️. It's probably not something watchtower itself can fix unfortunately.

[WKampel]

I added those lines to both frontend and backend, and I'm getting this error:

services.backend.labels contains {"com.centurylinklabs.watchtower.depends-on": "nginx"}, which is an invalid type, it should be a string
services.frontend.labels contains {"com.centurylinklabs.watchtower.depends-on": "nginx"}, which is an invalid type, it should be a string

Nginx is already setup to depend on frontend / backend. I don't understand why it doesn't automatically restart when those containers are changed.

[piksel]

Because that is a compose-only thing that watchtower wasn't written to handle. It has been proposed read depends-on information from the labels that docker-compose puts on the containers, but nothing has been implemented.

Regarding the error, are you combining array and map syntax perhaps? Use either - "foo=bar" or "foo": "bar" (ref)