Media storage

Options

• Brand-name AWS S3
• Linode (what I use — great company & simple pricing!)
• DigitalOcean (also great company & simple pricing)
• Vultr
• Cloudflare R2
• Backblaze B2
• Exoscale
• Wasabi (mixed reviews, especially re: unpredictable pricing, but I've never tried)
• ...any service with a fully S3 (or OpenStack Swift) compatible API

References

• https://github.com/cybrespace/cybrespace-meta/blob/master/s3.md
• https://thomas-leister.de/en/mastodon-s3-media-storage/
• https://stanislas.blog/2018/05/moving-mastodon-media-files-to-wasabi-object-storage/
• https://docs.joinmastodon.org/admin/config/#files
• https://docs.joinmastodon.org/admin/optional/object-storage-proxy/

Notes

Configuring your bucket so files are readable publicly (but only writable by Mastodon) is wildly different for every provider (some have a checkbox, some need a whole ACL). Make sure you check out their docs before you start using the new bucket.

Config

⚠️ Stop your server first (systemctl stop mastodon-*) so you don't end up with duplicate and/or missing files!

/home/mastodon/live/.env.production

For AWS, most defaults are fine — S3_BUCKET, S3_HOSTNAME, AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY should be all you need. You also need to set S3_REGION (not listed below).

This is my config for Linode Object Storage, which needs a few extra vars:

S3_ENABLED=true
S3_BUCKET=my-mastodon-storage-bucket
S3_PROTOCOL=https
S3_HOSTNAME=us-east-1.linodeobjects.com
S3_ENDPOINT=https://us-east-1.linodeobjects.com
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=

Migration

The following only applies if you currently have a Mastodon instance up-and-running and are only migrating to S3 now.

You should definitely set up everything above before starting a fresh Mastodon server for the first time — migrating existing media has many points of failure. But we (I) don't always do what we (I) should...

⚠️ Again, stop your server first (systemctl stop mastodon-*) so you don't end up with duplicate and/or missing files!

Install aws-cli

...even if you're not using AWS (see Linode's instructions)

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
aws --version

Configure aws-cli

You can use the same access key and secret from the ones you used in .env.production above.

# configure a separate profile for Linode.
# enter access key and secret from cloud.linode.com
# region: keep default (US) even if your bucket is in a different country
# output: keep default
aws configure --profile linode

# test by listing buckets
# note: this is where you'd swap out us-east-1 with your bucket's actual region
aws s3 ls --profile=linode --endpoint=https://us-east-1.linodeobjects.com

Move existing media

Until now, media has been saved go /home/mastodon/live/public/system — we need to move that entire folder to the root of the new bucket.

Depending on how long since your server has been online, this may take a looooong time.

# it's a good idea to delete old media from the local filesystem first
cd /home/mastodon/live
RAILS_ENV=production ./bin/tootctl media remove --days=7

# use aws-cli to copy everything to the new bucket
aws s3 sync public/system/ s3://my-mastodon-storage-bucket/ --endpoint-url=https://us-east-1.linodeobjects.com

Verification

This is for peace of mind, but optional. If anything minor went wrong, Mastodon should fix itself over time.

cd /home/mastodon/live
RAILS_ENV=production ./bin/tootctl media remove-orphans --fix-permissions
RAILS_ENV=production ./bin/tootctl media usage

Delete local files

ONLY ONCE YOU'RE SURE EVERYTHING IS WHERE IT SHOULD BE, AND NEW MEDIA ISN'T STILL BEING SAVED TO AND/OR SERVED FROM HERE!

cd /home/mastodon/live
rm -rf ./public/system