Sidekiq and media permissions

[justindthomas]

Hi there,

I moved my Mastodon service to a new instance and just cannot figure out what is going on with my media directories. The Mastodon services are run as mastodon and the directory structure is owned by mastodon:

mastodon@mastodon:~$ ps aux | grep mastodon
mastodon   919  0.3  1.0 588944 40704 ?        Ssl  21:16   0:00 /usr/bin/node ./streaming
mastodon   924  3.0  6.0 656856 240936 ?       Ssl  21:16   0:08 puma 5.3.2 (tcp://127.0.0.1:3000) [live]
mastodon   925 10.2  7.6 770340 308928 ?       Ssl  21:16   0:28 sidekiq 6.2.1 live [2 of 25 busy]
mastodon  1053  0.1  1.0 756404 42908 ?        Sl   21:16   0:00 /usr/bin/node /mnt/live/streaming
mastodon  1304  0.5  6.4 1402460 259528 ?      Sl   21:16   0:01 puma: cluster worker 0: 924 [live]
mastodon  1305  0.5  6.5 1402460 260956 ?      Sl   21:16   0:01 puma: cluster worker 1: 924 [live]

mastodon@mastodon:/mnt/live/public/system$ ls -la
total 20
drwxrwxr-x  5 mastodon mastodon 4096 Feb  6  2021 .
drwxrwxr-x 11 mastodon mastodon 4096 Sep 27 17:28 ..
drwxrwxr-x  4 mastodon mastodon 4096 Jan 26  2021 accounts
drwxrwxr-x  6 mastodon mastodon 4096 Dec  3  2020 cache
drwxrwxr-x  3 mastodon mastodon 4096 Oct  2 00:00 media_attachments

The filesystem is mounted rw:

ubuntu@mastodon:~$ mount
. . .
/dev/vdb on /mnt type ext4 (rw,relatime,stripe=2,data=ordered,x-systemd.requires=cloud-init.service)

But sidekiq just will not save avatars or media_attachments:

Oct 21 21:23:24 mastodon bundle[925]: 2021-10-21T21:23:24.154Z pid=925 tid=3aeh class=ActivityPub::ProcessingWorker jid=df583062e6704a9a2670fd6b elapsed=16.121 INFO: fail
Oct 21 21:23:24 mastodon bundle[925]: 2021-10-21T21:23:24.155Z pid=925 tid=3aeh WARN: {"context":"Job raised exception","job":{"retry":8,"queue":"default","backtrace":true,"class":"ActivityPub::ProcessingWorker","args":[9904,"{\"@context\": \"https://www.w3.org/ns/activitystreams\", \"type\": \"Announce\", \"to\": [\"https://mastodon-relay.thedoodleproject.net/followers\"], \"actor\": \"https://mastodon-relay.thedoodleproject.net/actor\", \"object\": \"https://nerdculture.de/users/ij/statuses/107141425057253649\", \"id\": \"https://mastodon-relay.thedoodleproject.net/activities/9902f5e8-77a7-4c63-8e92-e1017fc9695c\"}",null],"jid":"df583062e6704a9a2670fd6b","created_at":1634848415.6624618,"enqueued_at":1634851388.030432,"error_message":"Read-only file system @ dir_s_mkdir - /mnt/live/public/system/cache/accounts/avatars/107/039","error_class":"Errno::EROFS","failed_at":1634848425.6784348,"retry_count":6,"error_backtrace":"eJytzVEKwjAQBNCrhPxUQbJlQZReRaQau8Wlm2xJmoC3t+QM/ZxhePOw8NVA\nEN5500kjuOQpVqiUMmvMgO7mEIQ9pOJ/LfYws1DZWPK+HvDaDxzNKywTp85e\njhPnMh6I4r2hXvSzmBMaoUqSz2Yv28u4dvb5B8UPWs4=\n","retried_at":1634849987.9962938},"jobstr":"{\"retry\":8,\"queue\":\"default\",\"backtrace\":true,\"class\":\"ActivityPub::ProcessingWorker\",\"args\":[9904,\"{\\\"@context\\\": \\\"https://www.w3.org/ns/activitystreams\\\", \\\"type\\\": \\\"Announce\\\", \\\"to\\\": [\\\"https://mastodon-relay.thedoodleproject.net/followers\\\"], \\\"actor\\\": \\\"https://mastodon-relay.thedoodleproject.net/actor\\\", \\\"object\\\": \\\"https://nerdculture.de/users/ij/statuses/107141425057253649\\\", \\\"id\\\": \\\"https://mastodon-relay.thedoodleproject.net/activities/9902f5e8-77a7-4c63-8e92-e1017fc9695c\\\"}\",null],\"jid\":\"df583062e6704a9a2670fd6b\",\"created_at\":1634848415.6624618,\"enqueued_at\":1634851388.030432,\"error_message\":\"Read-only file system @ dir_s_mkdir - /mnt/live/public/system/cache/accounts/avatars/107/039\",\"error_class\":\"Errno::EROFS\",\"failed_at\":1634848425.6784348,\"retry_count\":6,\"error_backtrace\":\"eJytzVEKwjAQBNCrhPxUQbJlQZReRaQau8Wlm2xJmoC3t+QM/ZxhePOw8NVA\\nEN5500kjuOQpVqiUMmvMgO7mEIQ9pOJ/LfYws1DZWPK+HvDaDxzNKywTp85e\\njhPnMh6I4r2hXvSzmBMaoUqSz2Yv28u4dvb5B8UPWs4=\\n\",\"retried_at\":1634849987.9962938}"}
Oct 21 21:23:24 mastodon bundle[925]: 2021-10-21T21:23:24.155Z pid=925 tid=3aeh WARN: Errno::EROFS: Read-only file system @ dir_s_mkdir - /mnt/live/public/system/cache/accounts/avatars/107/039
Oct 21 21:23:24 mastodon bundle[925]: 2021-10-21T21:23:24.155Z pid=925 tid=3aeh WARN: /home/mastodon/.rbenv/versions/2.7.2/lib/ruby/2.7.0/fileutils.rb:250:in `mkdir'
Oct 21 21:23:24 mastodon bundle[925]: /home/mastodon/.rbenv/versions/2.7.2/lib/ruby/2.7.0/fileutils.rb:250:in `fu_mkdir'
Oct 21 21:23:24 mastodon bundle[925]: /home/mastodon/.rbenv/versions/2.7.2/lib/ruby/2.7.0/fileutils.rb:228:in `block (2 levels) in mkdir_p'

If I manually create each part of the directory as the mastodon user (e.g., mkdir /mnt/live/public/system/cache/accounts/avatars/107/039), it will continue with another message saying that the next level directory doesn't exist. I can then manually create that and it will eventually save the file once I get the whole directory structure created by hand. But obviously that's not sustainable.

I've searched and searched and can't find any instances of this problem elsewhere. Any clues or advice you could provide would be greatly appreciated!

-Justin

[Gargron]

I suspect it’s related to the systemd sandboxing rules. I’m not a specialist in those but I would remove them from Mastodon’s systemd service files, and see what happens after an appropriate restart.

[justindthomas]

Yep, that seems to have been the issue. I commented out this section of mastodon-sidekiq.service. I'll go back and try to re-enable some of the bits. Thanks!

# Sandboxing
#ProtectSystem=strict
#PrivateTmp=true
#PrivateDevices=true
#PrivateUsers=true
#ProtectHostname=true
#ProtectKernelLogs=true
#ProtectKernelModules=true
#ProtectKernelTunables=true
#ProtectControlGroups=true
#RestrictAddressFamilies=AF_INET
#RestrictAddressFamilies=AF_INET6
#RestrictAddressFamilies=AF_NETLINK
#RestrictAddressFamilies=AF_UNIX
#RestrictNamespaces=true
#LockPersonality=true
#RestrictRealtime=true
#RestrictSUIDSGID=true
#PrivateMounts=true
#ProtectClock=true

[ClearlyClaire]

Seems like this issue is documented in this (unmerged) PR: mastodon/documentation#892