[#6531] Store raw emails with ActiveStorage #6539
Conversation
gbp
force-pushed
the
6531-store-raw-emails-with-activestorage
branch
2 times, most recently
from
4e79c16
to
55efdc3
Compare
gbp
force-pushed
the
6531-store-raw-emails-with-activestorage
branch
5 times, most recently
from
45c6213
to
430f2b9
Compare
There was a problem hiding this comment.
Magic! 🧙 Couple of minor tweaks to address but all good in principle. Haven't run it with S3 backend but – I presume – one wouldn't really notice much change!
Probably want to coordinate when we merge this to make sure we've got the config files in place and a plan for migration.
db/migrate/20210928115500_create_active_storage_tables.active_storage.rb
Outdated
Show resolved
Hide resolved
|
@garethrees any other thoughts on the fix ups before I merge? |
gbp
force-pushed
the
6531-store-raw-emails-with-activestorage
branch
from
48fedd5
to
8153c03
Compare
gbp
force-pushed
the
6531-store-raw-emails-with-activestorage
branch
from
8153c03
to
1f16801
Compare
gbp
force-pushed
the
6531-store-raw-emails-with-activestorage
branch
from
8b20931
to
6a32693
Compare
There was a problem hiding this comment.
This all looks great! Such an improvement!
The performance is noticeably slower, obviously, and that might get passed on to users since it is possible for us to do a first-parse of a raw email during the request cycle. I expect many will be parsed before storage:promote happens, but they might not be. We did once proactively parse pro requests, so we could re-introduce something like that if it becomes a problem. I think we'll want to keep an eye on the amount of GET requests we're seeing, since the only other place we'd see a GET, I think, is via /admin/raw_emails/:id.format.
Defaults to disk storage but adds example config and the gems required to store in Amazon S3 (or S3 like services), Azure or Google Cloud Storage. Fixes #6530
Preparing for a new context when we move to a `ActiveStorage` backed file store.
When the `ActiveStorage::Blob` file isn't attached then use the old file store. There are no specs currently for the `#directory` and `#filepath` methods. This due to a `if Rails.env.test?` conditional in the model. These methods won't be used when a install switches to `ActiveStorage` and so they can be deprecated.
This won't be used in 0.42 as in 0.41 installs will be required to migrate to `ActiveStorage`.
Add tasks to: 1. mirror files from primary ActiveStorage backend to secondary backends 2. promote mirrored files to be served from the secondary backend 3. unlink primary file if mirrored and promoted to secondary backend.
gbp
force-pushed
the
6531-store-raw-emails-with-activestorage
branch
from
6a32693
to
f70328e
Compare
Relevant issue(s)
Fixes #6530
Fixes #6531
What does this do?
ActiveStoragesupport for raw emailsraketask to migrate raw emails on disk to aActiveStoragebacked file store.Why was this needed?
ActiveStorageis integrated with cloud based services to so file stores can exist on S3. This will help installations as on disk raw emails file stores can get quite large. This makes migrated servers difficult and limits the ability to run more than one application instance.Implementation notes
The application doesn't require migration straight away and will fall back to the old file store if the files aren't attached.