Dependent containers are not shutdown or restarted

[kje1]

Describe the bug
I recently commented on #504 (comment) which has now been closed as the suggested fix was to use the com.centurylinklabs.watchtower.depends-on label for dependent containers. I have done this but unfortuantely this is still not working for me. Today my vpn container updated and a service which uses the vpn container for it's network, was initially not shut down but also not restarted again after the vpn image was updated. To solve this I have to destroy the dependent container and recreate it as the container ID of the vpn changes after an update.

As you can see below, I have both depends_on and the com.centurylinklabs.watchtower.depends-on label setup on the dependent container but there is nothing in the logs indicating that watchtower did anything with this containers.

Am I using the label wrong by any chance?

To Reproduce
Steps to reproduce the behavior:

1. Container A with dependency link to container B
2. Watchtower updates container B

Expected behavior

1. Container A should be stoppped
2. Container B should be updated and restarted
3. Container A should be restarted and connect to new container B

Docker Compose
Extract from docker compose file...

#WatchTower
  watchtower:
   image: containrrr/watchtower
   hostname: watchtower
   container_name: watchtower
   restart: always
   volumes:
    - /var/run/docker.sock:/var/run/docker.sock
   command: --interval "3600" --cleanup --debug
   environment:
    - TZ=${TZ}

#vpn        
  vpn:
    image: dperson/openvpn-client
    container_name: vpn
    # cap_add, security_opt, and volume required for the image to function
    cap_add:
      - net_admin
    environment:
      - TZ=Europe/London
      - FIREWALL=""
    networks:
      - default
    dns:
      - 8.8.8.8
      - 8.8.4.4
    read_only: true
    tmpfs:
      - /tmp
      - /run
    restart: always
    security_opt:
      - label:disable
    stdin_open: true
    tty: true
    volumes:
      - /dev/net:/dev/net:z
      - ${USERDIR}/docker/vpn:/vpn
    ports:
      - "XXXX:XXXX"

#Ombi
  ombi:
   image: linuxserver/ombi
   container_name: ombi
   restart: always
   volumes:
    - ${USERDIR}/docker/ombi:/config
    - ${USERDIR}/docker/shared:/shared
   environment:
    - PUID=${PUID}
    - PGID=${PGID}
    - TZ=${TZ}
   network_mode: service:vpn
   depends_on:
    - vpn
   labels:
    - "com.centurylinklabs.watchtower.depends-on=vpn"

Logs from running watchtower with the --debug option

time="2020-07-07T15:00:25+01:00" level=info msg="Stopping /vpn (bef6f210ff074c07efcce051ca1980f12bf1b4eddc5e6a16800b9d78b45de6c9) with SIGTERM"
time="2020-07-07T15:00:31+01:00" level=debug msg="Removing container bef6f210ff074c07efcce051ca1980f12bf1b4eddc5e6a16800b9d78b45de6c9"
time="2020-07-07T15:00:31+01:00" level=info msg="Creating /vpn"
time="2020-07-07T15:00:31+01:00" level=debug msg="Starting container /vpn (1acd36abab38d472c93b26c8d914a42a94f5ad25fde70478866721286814a368)"
time="2020-07-07T15:00:32+01:00" level=info msg="Removing image sha256:2403135f134b336da433a70a127d88f24a831463ad7c17a2ad1a3bbe1c376226"
time="2020-07-07T15:00:32+01:00" level=debug msg="Scheduled next run: 2020-07-07 16:00:01 +0100 BST"

[github-actions]

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

[piksel]

I think you need to specify the label as:

   labels:
     com.centurylinklabs.watchtower.depends-on: vpn

[kje1]

Is this really the case? I've got labels specified like that for containers that I proxy via traefik that work as expected using my syntax. The label also appears in portainer as I would expect as well.

[piksel]

Maybe not, but it looked like it might be unintentional, as it would be parsed as an array of strings, rather than a key/value map, but I guess it's smart enough to detect that. Also, I made mistake in the syntax above. There shouldn't be any dashes before map items.

[kje1]

Hopefully it is just a syntax issue rather than the label not actually working correctly.

I just double checked all of my traefik labels and what I originally said was slightly wrong. It seems some of them have the syntax that you wrote and some have the syntax I wrote, as below. I must admit, I am fairly new to all of this and followed a pretty comprehensive guide when setting up my proxy with labels so I don't know much about how the syntax should be for strings or key / value map. What I can see though is that all of my traefik labels have the dash before them but your suggestion is to not include that?

Example traefik labels - 1 with equals, one with colon;

labels
  - "traefik.frontend.headers.STSPreload=true"
  - "traefik.frontend.headers.customFrameOptionsValue: allow-from https:${DOMAINNAME}"

The documentation here doesn't give any indication on how to use the label in other containers so for a novice like myself it has been a case of trial and error, but unforutantely with no success.

So based on all of that, how should the label be written? With or without a dash? With a colon instead of an equal sign? Does it need to be encompassed in quotation marks? I'll give your original suggestion a try too and hopefully that will fix the issue I'm having.

[piksel]

In this case, labels is technically an array of strings, equivalent of the following json:

"labels": [
  "traefik.frontend.headers.STSPreload=true",
  "traefik.frontend.headers.customFrameOptionsValue: allow-from https:${DOMAINNAME}"
]

compared to the key/value object:

"labels": {
  "traefik.frontend.headers.STSPreload": true,
  "traefik.frontend.headers.customFrameOptionsValue": "allow-from https:${DOMAINNAME}"
}

But I think docker-compose parses them and splits them to key-values, but I couldn't find it in the documentation. I have seen other instance of this in docker-compose files, so both probably work.
But you could try changing the label to:

    com.centurylinklabs.watchtower.depends-on: vpn

and see if it makes a difference, it's the only thing I found that stood out. It looks like it should work when looking through the code,

I'm thinking that Watchtower should probably output in the log why it's restarting containers, maybe a note about how many dependencies it found for a updated container? It really is quite hard to tell what is happening in these cases right now.

[simonaronsson]

@Saicheg might be able to verify this, but from a) looking at the code and b) knowing that what it tries to emulate is links, I think this feature works in the opposite direction of what you've set up.

Depends on, from what I can tell, does not include a list of containers that you depend on, but rather a list of containers that depend on you. In your case, if ombi is updated, It would also restart vpn. A more obvious naming of the label would likely have been com.centurylinklabs.watchtower.children, which must have blown past me while reviewing it.

Could you please try moving your label to the VPN container instead and specify ombi as the value?

Thanks.

[kje1]

I came to a very similar conclusion yesterday actually. I'll reverse the dependencies and let you know if that solves the problem. Thanks!

[kje1]

Hi,

I've added the label like this as suggested. Could you confirm that this is the correct syntax please? Unfortunately it's now just a case of waiting for the the vpn to update to see if this works correctly. I guess I could try adding a depends on to another container as a trial...

I think @piksel makes a very good point above. Something in the logs to show what dependencies exist for a particular container would be really useful. That would give confidence during the inital setup process that everything should work as intended when containers do update.

#vpn        
  vpn:
    image: dperson/openvpn-client
    container_name: vpn
    # cap_add, security_opt, and volume required for the image to function
    cap_add:
      - net_admin
    environment:
      - TZ=Europe/London
      - FIREWALL=""
    networks:
      - default
    dns:
      - 8.8.8.8
      - 8.8.4.4
    read_only: true
    tmpfs:
      - /tmp
      - /run
    restart: always
    security_opt:
      - label:disable
    stdin_open: true
    tty: true
    volumes:
      - /dev/net:/dev/net:z
      - /home/kje1/docker/vpn:/vpn
    ports:
      - "XXXX:XXXX"
      - "XXXX:XXXX"
      - "XXXX:XXXX"
      - "XXXX:XXXX"
      - "XXXX:XXXX"
      - "XXXX:XXXX"
    labels:
      - "com.centurylinklabs.watchtower.depends-on: jackett, radarr, sonarr, qbittorrent, tautulli, ombi"

[simonaronsson]

    labels:
      - "com.centurylinklabs.watchtower.depends-on: jackett, radarr, sonarr, qbittorrent, tautulli, ombi"

I would write it as:

labels:
  com.centurylinklabs.watchtower.depends-on: jackett,radarr,sonarr,qbittorrent,tautulli,ombi

just to remove any ambiguity regarding how it will be parsed. Other than that, yeah - this should work.

[zethis]

Same problem for me, depends_on: works perfectly with docker-compose.
Why not with watchtower ?

Edit:
It appears depends_on: it's only available with docker-compose, not with docker

Edit2:
Issue talking about that here: docker/compose#5608
Not merged, Closed.

So if Watchtower use docker run after update, it's logical 👍

[xAlstrat]

I can't make it work. I think I tried all the possible options:

    labels:
      com.centurylinklabs.watchtower.depends-on: "python" # Also tried without the quotes (")

And the string version:

    labels:
       - "com.centurylinklabs.watchtower.depends-on=python"

I also added the label at build time in Dockerfile for labeling the image:

LABEL com.centurylinklabs.watchtower.depends-on="python"

The command docker inspect shows that the label is set correctly (for both image and container), so I think there is no problem with the label format. Here is my docker inspect raspbian (that depends on python container/service):

            ...
            "Labels": {
                "com.centurylinklabs.watchtower.depends-on": "python",
                "com.docker.compose.config-hash": "83fd007a7de0d61d8576236c62e335195d0f978fff8e85228a70c5d9f0b515e1",
                ...

This is my docker-compose.yml:

version: '3.7'

services:
  python:
    image: my-repo/python:latest
    restart: always
    container_name: python
    labels:
      com.centurylinklabs.watchtower.depends-on: raspbian
  raspbian:
    image: my-repo/raspbian:latest
    restart: always
    depends_on:
      - python # Obviously this doesn't work
    container_name: raspbian
    labels:
      com.centurylinklabs.watchtower.depends-on: python
  watchtower:
    restart: always
    image: containrrr/watchtower
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./config.json:/config.json
    command: --interval 30 --cleanup --include-stopped --revive-stopped

When Watchtower detects a new python image, raspbian isn't stopped and logs shows no error about that container.

I even tried using depends-on label in python instead of raspbian (as you can see in the docker-compose file) like a kind of 'those-depends-on-me' label, but didn't work.

I'm running Watchtower and other services with docker-compose up.

UPDATE:

For some reason, I think my Watchtower is handling container names with a slash '/' as preffix (as shown in the logs in a previous post: /vpn), docker inspect is also showing my container name as /python. So I tried using "/python" instead of "python", but didn't work.

The interesting thing was when I configured both containers to depends-on the other:

  python:
    ...
    container_name: python
    labels:
      com.centurylinklabs.watchtower.depends-on: "/raspbian"
  raspbian:
    ...
    container_name: raspbian
    labels:
      com.centurylinklabs.watchtower.depends-on: "/python"

Using '/' preffix, Watchtower detected a circular dependency showing it in the logs:

watchtower_1_c63e785908fe | time="2020-07-31T17:18:03Z" level=info msg="circular reference to /raspbian"

When a new 'python' imagen is detected, the logs shows that a new image was found but 'python' container is not getting stopped/updated (neither 'raspbian' container). It seems ok for me because is there a circular reference.

So, using slashes, Watchtowers detects circular references but no unidirectional references (I tried referencing from python->raspbian and raspbian->python when pushing a new python image).