Permission update on docker entrypoint takes a long time #3194
Comments
|
The goal is not to chown So if I understand well, this commandes takes a long time for you? |
|
I'm having the same issue, here's some info:
|
|
No problem with : |
|
Only thing I see is the storage driver, and indeed on my desktop computer which has btrfs:
|
|
I tried on my Mac, which is using |
|
Yeah so I guess we can accurately say that this is not an issue with Mastodon, but that it's linked to the Docker storage driver choice. |
|
I agree with @fmauNeko, unfortunately we can't do more here. cc @xataz if you have an idea. |
|
You are right, took 9m20s to update permissions on my cloud server (overlay2 storage driver) and 7s on my local machine (aufs storage driver). I thought find command would cost more, but its not the case. Looks like a storage driver issue. I'll look into improving my cloud docker setup. |
|
I tried to change the storage driver from overlay2 to aufs on my debian jessie VC1S scaleway instance, but the docker daemon fail to start, my kernel dosen't support aufs. |
|
It's weird to have such bad performance on overlay2, which should be better than aufs. I'm also running on scaleway (VC1M). |
|
Ok i can overwrite the entrypoint script : https://docs.docker.com/compose/compose-file/#entrypoint |
|
@katarpilar I believe would be best to figure out why we have such bad performance on our cloud setup and add the solution to Troubleshoot docs. I assume a lot of admins use scaleway services. |
|
FYI I just got 13s run on another aufs non-ssd cloud service. |
|
That's the reason why I gave up overlay2. The performances are terrible, not with this command in particular, but in general. That being said
Speaking of OverlayFS, I noticed this performance issue a while ago. I came up with a hack which consists of refreshing (somehow) the files in the layers (so basically this could fix your issue), but it stopped working with a Docker update. |
|
Well I have bad performance with aufs myself, so that's strange. |
|
Same issue (overlay2) : |
|
I am also having HORRIBLE load times with permissions running locally on macOS. 9+ minutes |
|
@malicioustoker is it overlay2 storage driver? you can get the information with |
|
Yeah, it is using Overlay2
Containers: 12
Running: 6
Paused: 0
Stopped: 6
Images: 56
Server Version: 17.03.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.27-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.65 GiB
Name: moby
ID: JN5A:LMHV:2GUL:FGKP:GTOH:5IWI:V3TP:GTUD:TEWR:5RBY:EWZ3:U6UY
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 64
Goroutines: 63
System Time: 2017-05-22T15:10:42.611514466Z
EventsListeners: 1
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
…Sent from my iPhone
On May 22, 2017, at 7:47 AM, Miguel Peixe ***@***.***> wrote:
@malicioustoker is it overlay2 storage driver? you can get the information with docker info command.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
How can I change it from Overlay2 to something else? That's just the default option when Docker is installed on macOS |
|
@malicioustoker Interesting, I still have But you can change your storage-driver easily :
I really think we should open an issue at Docker (moby/moby) rather than arguing about this change, because what else can we do? I do understand your frustration if it's taking too long (10 minutes ??? Come on!), and I suffered from this bug during months before finally giving up |
|
@waterfall trying that now - thanks 😊 I also sent you an email yesterday - do you have time to chat about something. You're the only other person I know running Mastodon off macOS
…Sent from my iPhone
On May 22, 2017, at 8:51 AM, Wonderfall ***@***.***> wrote:
@malicioustoker Interesting, I still have aufs on my machine (that said I don't have the latest version yet).
But you can change your storage-driver easily :
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
For Scaleway users, this is how you change to aufs:
{
"storage-driver": "aufs"
}Restart docker service and that's it. |
|
Changing to aufs fixed the issue - it now takes no more than 5 seconds to change permissions - thanks everyone! |
|
Can someone try the For this to work you'll have to enable experimental features in Docker, put this in |
|
What's the difference between Overlay2 and the other file system? If Overlay2 is /suppose/ to be better, I'll switch back and test this command out for you if you'd like
…Sent from my iPhone
On May 24, 2017, at 7:30 AM, Wonderfall ***@***.***> wrote:
Can someone try the --squash option ? Someone still using overlay2 I mean.
docker build --squash -t mastodon .
For this to work you'll have to enable experimental features in Docker, put this in /etc/docker/daemon.json :
{
"experimental": true
}
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
OverlayFS implements union mount, it's supposed to be faster, and it's in Linux kernel upstream. It will overtake That said there are other alternatives :
|
|
If you need to setup a new server for Docker though, use |
|
im on overlay2 too and it takes over 30 minutes for me. im all morning for upgrading because the recreate and migrate commands start chown in the entrypoint. |
|
@xsteadfastx No you're not stuck. You can still use
|
|
@fmauNeko yep... installed... |
|
@xsteadfastx try installing |
|
@xsteadfastx also, don't forget to add to |
|
@miguelpeixe no kernel support for aufs on ubuntu 16.04 |
|
Hum, that's weird, I'm using aufs on all my 16.04 servers, including my local setup. |
|
@xsteadfastx |
|
ok i have to say sorry... modprobe aufs after installing linux-image-extra-virtual did the trick... sorry for this discussion about the docker image... it works pretty well i was just in a bad mood because it tooks hours to upgrade mastodon. thanks for all the help. |
|
There seems to be no "fix" for this issue yet and some us don't have aufs compatible kernels or the luxury of creating a VM or attaching an extra partition with brtfs or zfs for the sake of prototyping. So my question here would be, how harmful exactly is just doing:
instead of: Would it suffice to just add a check to the script and when overlay or overlay2 is detected run a warning and just chown the entire system directory ? It seems like a good compromise considering the speed "bug" is with docker (or overlay2... depending on how you think about it) and may not be fixed in a while, however docker is slowly migrating to overlay and linunx distros are slowly removing aufs support out of the default shipped kernels. My current fix is to run the mastodon instance with the original chown (as to not risk any issues) and "manually" modify the script inside the image to the chown of the directory for running any other tasks (e.g. creating admin users) much faster. But that is hardly the most convenient think to do, since it require an file edit every time I want to reboot my instance. |
|
Well it won't be recursive, and I'm not sure chown -R would be faster than the current solution. |
|
Just tried it on my 2 CPU, 8 GB RAM server and it look 49 mins... 😦 |
|
What do you think about For me it would be a nice workaround. |
|
Not sure how big of an issue it is, but while running this commit, I now get this error
Step 16/19 : COPY --chown=${UID}:${GID} . /mastodon
ERROR: Service 'web' failed to build: unable to convert uid/gid chown string to host mapping: can't find uid for user ${UID}: no such user: ${UID}
…-- Reverend Glen
On Tuesday, Feb 20, 2018 at 8:25 AM, Eugen Rochko ***@***.*** ***@***.***)> wrote:
Closed #3194 (#3194) via #6514 (#6514).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub (#3194 (comment)), or mute the thread (https://github.com/notifications/unsubscribe-auth/ABXY3gH-U_w3RUgjmLjQLVCzqkDJIpOIks5tWvHugaJpZM4NheBb).
|
|
The solution here would be to use the user/group names instead of the variables. I'll come up with a PR. |
|
@malicioustoker This should be fixed now. |
|
Can confirm that it has been fixed, thanks!
…-- Reverend Glen
On Tuesday, Feb 20, 2018 at 10:14 AM, Moritz Heiber ***@***.*** ***@***.***)> wrote:
@malicioustoker (https://github.com/malicioustoker) This should be fixed now.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub (#3194 (comment)), or mute the thread (https://github.com/notifications/unsubscribe-auth/ABXY3iOji2jL1gAsLBeH0_mfj8DY4l-aks5tWwtmgaJpZM4NheBb).
|
With the current approach on docker entrypoint for updating the files to the new custom UID/GID it takes forever to finish the process, which timeouts in a reasonable production container health check.
Why not just use
chown -rf mastodon:mastodon /mastodon/public/systeminstead of finding and filtering non-mastodon files?master(If you're a user, don't worry about this).The text was updated successfully, but these errors were encountered: