tmpfs unusable after container restart

[tmuncks]

Using tmpfs does not work properly, after container restart.

version: '3.1'
services:
  varnish:
    image: "varnish:latest"
    restart: "always"
    volumes:
      - "./varnish.vcl:/etc/varnish/default.vcl:ro"
    tmpfs:
      - "/var/lib/varnish/varnishd:exec"

When the /var/lib/varnish/varnishd path already exists, docker does strange things when mounting a tmpfs. The end result is:

First run:

drwxrwxrwt 2 root    root      40 Apr 24 00:40 varnishd

After docker-compose restart varnish:

drwxr-xr-x 2 root    root      40 Apr 24 00:40 varnishd

This in turn gives:

varnish_1    | Error: Cannot create test-file in /var/lib/varnish/varnishd (Permission denied)
varnish_1    | Check permissions (or delete old directory)
varnish_1    | (-? gives usage)

This has not been an issue previously, so my suspicions go to the USER varnish statement that was recently added near the bottom of the Dockerfile?

[tmuncks]

Just double checked that older builds (e.g. 6.6 and 7.0.1) does not exhibit this problem.

[gquintard]

hi @tmuncks, can you detail how to reproduce this please? I used you yaml file, ran docker-compose up and in another terminal ran docker-compose restart, and I get the container stuck in the restarting state (which isn't great, but also happens on 7.0.1), but I don't get the error message you shared

[tmuncks]

I think you are precisely experiencing this problem. The actual messages are in the log:

docker-compose logs varnish

For good measure, this is how I'm able to reproduce 100% of the time:

root@example:/docker/test# docker-compose up -d --force-recreate
Recreating test_varnish_1 ... done

root@example:/docker/test# docker-compose logs varnish
Attaching to test_varnish_1
varnish_1  | Debug: Version: varnish-7.1.0 revision c2c07517ddddbc187604b800eff733296e93aaad
varnish_1  | Debug: Platform: Linux,5.15.0-25-generic,x86_64,-jnone,-smalloc,-sdefault,-hcritbit
varnish_1  | Debug: Child (20) Started
varnish_1  | Child launched OK
varnish_1  | Info: Child (20) said Child starts

root@example:/docker/test# docker-compose restart varnish
Restarting test_varnish_1 ... done

root@example:/docker/test# docker-compose logs varnish
Attaching to test_varnish_1
varnish_1  | Debug: Version: varnish-7.1.0 revision c2c07517ddddbc187604b800eff733296e93aaad
varnish_1  | Debug: Platform: Linux,5.15.0-25-generic,x86_64,-jnone,-smalloc,-sdefault,-hcritbit
varnish_1  | Debug: Child (20) Started
varnish_1  | Child launched OK
varnish_1  | Info: Child (20) said Child starts
varnish_1  | Error: Manager got SIGTERM
varnish_1  | Debug: Stopping Child
varnish_1  | Info: Child (20) ended
varnish_1  | Info: Child (20) said Child dies
varnish_1  | Debug: Child cleanup complete
varnish_1  | Info: manager stopping child
varnish_1  | Info: manager dies
varnish_1  | Error: Cannot create test-file in /var/lib/varnish/varnishd (Permission denied)
varnish_1  | Check permissions (or delete old directory)
varnish_1  | (-? gives usage)
varnish_1  | Error: Cannot create test-file in /var/lib/varnish/varnishd (Permission denied)
varnish_1  | Check permissions (or delete old directory)
varnish_1  | (-? gives usage)
varnish_1  | Error: Cannot create test-file in /var/lib/varnish/varnishd (Permission denied)
varnish_1  | Check permissions (or delete old directory)
varnish_1  | (-? gives usage)
varnish_1  | Error: Cannot create test-file in /var/lib/varnish/varnishd (Permission denied)
varnish_1  | Check permissions (or delete old directory)
varnish_1  | (-? gives usage)

[gquintard]

thanks, that's useful. The permission change here is really hurting us...

The easy fix is to add user: root to your file:

version: '3.1'
services:
  varnish:
    image: "varnish:latest"
    restart: "always"
    volumes:
      - "./varnish.vcl:/etc/varnish/default.vcl:ro"
    tmpfs:
      - "/var/lib/varnish/varnishd:exec"
    user: root

But of course, this loses you the unprivileged behavior, which is annoying.

I need some time to understand why docker-compose changes the permissions

[tmuncks]

Cool... The user hack is certainly better for my setup, than not knowing if the service will survive the night.

The problem is actually a known docker issue: docker/for-linux#138 - unfortunately not one that appear to be solved anytime soon.

The problem - as I understand it - is that when mounting a tmpfs, the resulting permissions are lifted from the underlying mountpoint, which is super weird and not the way mount normally works. So it is not a big surprise that so many people (myself included) are slightly baffled the first time the come across this.

[gquintard]

thanks, I opened an issue on the compose github, but it looks like it goes even deeper than that. Welp, looks like there isn't much we can do about it then... I'll close the issue but will monitor the upstream issues.

Anyway, thanks for the clear report and extra info, that's definitely appreciated.

[gquintard]

@tmuncks, I'm guessing your comment didn't exactly end up the way you expected, can you check it and possibly edit it on GitHub please?

[tmuncks]

Sorry! I have no idea what that was... It was an accidental resend of a message I sent on April 24. I have deleted the bad one. Sorry again.

[CHallski]

Noting in case anyone else stumbles across this later.
There's a pretty good dive on docker/compose#9415 and the underlying moby/moby#20437, but if you create the directory & required permissions in your initial image (RUN mkdir -p -m 1777 /var/lib/varnish/varnishd), then the tmpfs permissions will be consistent afterwards (without the use of user:root as a workaround)

Thanks @gquintard for the upstream reporting.