Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add separate cache directory for non-local uploads #12821

Merged
merged 1 commit into from
Apr 26, 2020

Conversation

Gargron
Copy link
Member

@Gargron Gargron commented Jan 11, 2020

The following resources will be split into a top-level cache directory:

  • custom emojis
  • avatars
  • headers
  • media attachments
  • preview cards

...when they don't originate from local accounts. With the exception of preview cards, which are always counted as not local. The change comes into effect for anything uploaded or downloaded after the upgrade, old files remain in place, so as to not incur massive object storage costs moving potentially terabytes of data.

Adds tootctl upgrade storage-schema to upgrade all existing files at once.

@Gargron Gargron added the work in progress Not to be merged, currently being worked on label Jan 11, 2020
@Gargron Gargron marked this pull request as ready for review April 17, 2020 13:49
@Gargron Gargron removed the work in progress Not to be merged, currently being worked on label Apr 17, 2020
@ClearlyClaire ClearlyClaire self-requested a review April 17, 2020 15:23
@Gargron Gargron force-pushed the fix-media-cache-prefix branch 2 times, most recently from fb2a573 to d073d53 Compare April 17, 2020 20:41
Copy link
Contributor

@ClearlyClaire ClearlyClaire left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a very welcome change overall, although I have some minor concerns (see inline comments).

lib/paperclip/attachment_extensions.rb Show resolved Hide resolved
lib/mastodon/upgrade_cli.rb Outdated Show resolved Hide resolved
@Gargron Gargron force-pushed the fix-media-cache-prefix branch 3 times, most recently from 5203795 to a19fb36 Compare April 19, 2020 20:58
lib/mastodon/upgrade_cli.rb Outdated Show resolved Hide resolved
lib/mastodon/upgrade_cli.rb Show resolved Hide resolved
lib/mastodon/upgrade_cli.rb Outdated Show resolved Hide resolved
lib/mastodon/upgrade_cli.rb Show resolved Hide resolved
lib/mastodon/upgrade_cli.rb Show resolved Hide resolved
lib/mastodon/upgrade_cli.rb Outdated Show resolved Hide resolved
@Gargron Gargron merged commit c3ca380 into master Apr 26, 2020
@Gargron Gargron deleted the fix-media-cache-prefix branch April 26, 2020 22:24
zunda added a commit to zunda/mastodon that referenced this pull request Apr 27, 2020
@zunda
Copy link
Contributor

zunda commented Apr 27, 2020

This seems to have broken path to existing icons and custom emojis. E.g., my avatar which has been t https://s3.amazonaws.com/zundan-mastodon/accounts/avatars/000/000/001/original/89de63b40ca41c7e.gif is now rendered as https://s3.amazonaws.com/zundan-mastodon/system/accounts/avatars/000/000/001/original/89de63b40ca41c7e.gif_prefixaccounts/avatars/000/000/001/original/89de63b40ca41c7e.gif

git revert c3ca3801f fixed the problem for now. Not sure if related or not, my server is on ruby-2.7.1, makes direct use of S3 in us-east-1.

@mayaeh
Copy link
Contributor

mayaeh commented Apr 27, 2020

This seems to have broken path to existing icons and custom emojis.

This issue also occurs in my environment using Ruby 2.6.6 with OpenStack Swift.
This issue doesn't seem to occur on instances that are not using object storage.

@mayaeh
Copy link
Contributor

mayaeh commented Apr 27, 2020

This seems to fix it, but I don't understand why this problem occurs...

@orlea
Copy link
Contributor

orlea commented May 14, 2020

Is it the same as the old issue below?
#6675
#12382

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants