Permission update on docker entrypoint takes a long time

[miguelpeixe]

With the current approach on docker entrypoint for updating the files to the new custom UID/GID it takes forever to finish the process, which timeouts in a reasonable production container health check.

Why not just use chown -rf mastodon:mastodon /mastodon/public/system instead of finding and filtering non-mastodon files?

---

• I searched or browsed the repo’s other issues to ensure this is not a duplicate.
• This bug happens on a tagged release and not on master (If you're a user, don't worry about this).

[Gargron]

@Wonderfall

[Wonderfall]

The goal is not to chown /mastodon/public/system. It would take a long time to do that (believe me, I tried every combination possible). So find won't even go there with -path path -prune -o (-not -path path will exclude it but it will go there, so it will take time), it doesn't take time since it updates all permissions but not /public/system which likely contains a lot of data.

So if I understand well, this commandes takes a long time for you?
Can you run something like time docker run -ti --rm mastodon true?
Can you send me your docker info?

[fmauNeko]

I'm having the same issue, here's some info:
docker info

Containers: 53
 Running: 53
 Paused: 0
 Stopped: 0
Images: 130
Server Version: 17.05.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 477
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.8.0-46-generic
Operating System: Ubuntu 17.04
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.66GiB
Name: concorde.dissidence.ovh
ID: UEKG:ZQYV:I6EF:VIMY:TV5W:HXRD:SEDE:GPHJ:LKUV:OFGB:NQDY:VVNM
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: fmauneko
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

time docker run --rm -it gargron/mastodon:v1.4rc2 true

Creating mastodon user (UID : 991 and GID : 991)...
Updating permissions...
Executing process...
docker run --rm -it gargron/mastodon:v1.4rc2 true  0.01s user 0.01s system 0% cpu 1:29.87 total

[Wonderfall]

No problem with :

# docker info
Containers: 34
 Running: 34
 Paused: 0
 Stopped: 0
Images: 51
Server Version: 17.05.0-ce
Storage Driver: btrfs
 Build Version: Btrfs v4.7.3
 Library Version: 101
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.11.1
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.16GiB
Name: drogon
ID: TGPJ:KNGK:XV7N:LHP3:BXZG:AHLJ:QOR3:AL6F:ZHME:LHFZ:YBRP:MT4M
Docker Root Dir: /docker
Debug Mode (client): false
Debug Mode (server): false
Username: wonderfall
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: true

# time docker run -ti --rm mastodon true
Creating mastodon user (UID : 991 and GID : 991)...
Updating permissions...
Executing process...
docker run -ti --rm mastodon true  0.01s user 0.01s system 0% cpu 5.303 total

[fmauNeko]

Only thing I see is the storage driver, and indeed on my desktop computer which has btrfs:
docker info

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 15
Server Version: 17.05.0-ce
Storage Driver: btrfs
 Build Version: Btrfs v4.10.2
 Library Version: 102
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.10.13-1-ARCH
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.55GiB
Name: izanami
ID: LY2X:EOSA:YURV:H2OK:MVIU:HWPB:DXCX:GTUJ:DDKJ:2CUC:IZZT:RNFA
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

time docker run --rm -it gargron/mastodon:v1.4rc2 true

Creating mastodon user (UID : 991 and GID : 991)...
Updating permissions...
Executing process...
docker run --rm -it gargron/mastodon:v1.4rc2 true  0,02s user 0,02s system 0% cpu 12,941 total

[Wonderfall]

I tried on my Mac, which is using aufs, it takes around 5 seconds. Perhaps it's because of the SSD. But aufs is clearly less performant than btrfs.

# time find /mastodon -path /mastodon/public/system -prune -o -not -user mastodon -not -group mastodon -print0 | xargs -0 chown -f mastodon:mastodon
real	0m 4.26s
user	0m 0.10s
sys	0m 0.45s

# I ran another container to try a "non-optimised" command
# time chown -R 991:991 *
real	0m 5.45s
user	0m 0.18s
sys	0m 1.55s

[fmauNeko]

Yeah so I guess we can accurately say that this is not an issue with Mastodon, but that it's linked to the Docker storage driver choice.

[Wonderfall]

I agree with @fmauNeko, unfortunately we can't do more here.
If you're curious, this article also explains the reasons of why we should use an entrypoint rather than hardcode something in the Dockerfile : https://denibertovic.com/posts/handling-permissions-with-docker-volumes/ (just found it but it seems we had exactly the same idea! that's also what I'm doing for all my images you can find at wonderfall/dockerfiles)

cc @xataz if you have an idea.

[miguelpeixe]

You are right, took 9m20s to update permissions on my cloud server (overlay2 storage driver) and 7s on my local machine (aufs storage driver). I thought find command would cost more, but its not the case. Looks like a storage driver issue. I'll look into improving my cloud docker setup.

[katarpilar]

I tried to change the storage driver from overlay2 to aufs on my debian jessie VC1S scaleway instance, but the docker daemon fail to start, my kernel dosen't support aufs.
I will build my image without the chown command in the entrypoint script.
But i think this is better to tell admins to do the command before pulling new images in a release notes than to force all admins with overlayfs to build they own images if they dont want to wait 30 min (yes with 3 conteiners it takes time) to start their containers.

[miguelpeixe]

It's weird to have such bad performance on overlay2, which should be better than aufs. I'm also running on scaleway (VC1M).

[katarpilar]

Ok i can overwrite the entrypoint script : https://docs.docker.com/compose/compose-file/#entrypoint
Maybe we should add this in the documentation?

[miguelpeixe]

@katarpilar I believe would be best to figure out why we have such bad performance on our cloud setup and add the solution to Troubleshoot docs. I assume a lot of admins use scaleway services.

[miguelpeixe]

FYI I just got 13s run on another aufs non-ssd cloud service.

[Wonderfall]

That's the reason why I gave up overlay2. The performances are terrible, not with this command in particular, but in general.

That being said aufs is still the standard storage-driver for Docker so I thought no one would complain (but I was wrong :sad: ). Note that overlayfs shouldn't be really recommended for production environments, despite it has been seen as a potential successor to aufs :

As promising as OverlayFS is, it is still relatively young. Therefore caution should be taken before using it in production Docker environments.

Source : Docker documentation

Speaking of OverlayFS, I noticed this performance issue a while ago. I came up with a hack which consists of refreshing (somehow) the files in the layers (so basically this could fix your issue), but it stopped working with a Docker update.