Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Track Active Storage variants in the database #37901

Merged

Conversation

@georgeclaghorn
Copy link
Member

georgeclaghorn commented Dec 6, 2019

Currently, when a variant is requested, we check whether it exists in the service. If it doesn’t, we generate it on-demand and store it for reuse.

The existence check can add unacceptable latency to variant serving. We’ve seen existence checks to third-party services take upwards of 500ms in normal circumstances. Further, checking whether an object exists at a particular key before uploading it triggers eventual consistency in S3:

Amazon S3 provides read-after-write consistency for PUTS of new objects in your S3 bucket in all Regions with one caveat. The caveat is that if you make a HEAD or GET request to the key name (to find if the object exists) before creating the object, Amazon S3 provides eventual consistency for read-after-write.

This means that when a variant is requested for the first time, generated, and stored in S3, subsequent redirects to the variant’s service URL can fail indefinitely thereafter. The result is sporadically broken images.

This PR addresses the above concerns by doing away with the existence check. When a variant is requested for the first time, we generate it, store it, and track it in the application database. Now we can know whether we’ve already generated a variant without making a remote call to the storage service.

@georgeclaghorn georgeclaghorn force-pushed the georgeclaghorn:activestorage-variant-tracking branch 2 times, most recently from e48a277 to a0bcee3 Dec 6, 2019
@georgeclaghorn georgeclaghorn force-pushed the georgeclaghorn:activestorage-variant-tracking branch from a0bcee3 to 7217ee3 Dec 6, 2019
@rails-bot rails-bot bot added the docs label Dec 6, 2019
@georgeclaghorn georgeclaghorn marked this pull request as ready for review Dec 6, 2019
@georgeclaghorn georgeclaghorn force-pushed the georgeclaghorn:activestorage-variant-tracking branch from 7217ee3 to 1d2aecb Dec 6, 2019
@georgeclaghorn georgeclaghorn merged commit 7d0327b into rails:master Dec 6, 2019
2 checks passed
2 checks passed
build
Details
buildkite/rails Build #65390 passed (15 minutes, 7 seconds)
Details
@georgeclaghorn georgeclaghorn deleted the georgeclaghorn:activestorage-variant-tracking branch Dec 6, 2019
@georgeclaghorn georgeclaghorn added this to the 6.1.0 milestone Dec 6, 2019
activestorage/CHANGELOG.md Show resolved Hide resolved

def create_or_find_record(image:)
@record =
ActiveRecord::Base.connected_to(role: ActiveRecord::Base.writing_role) do

This comment has been minimized.

Copy link
@jeremy

jeremy Dec 6, 2019

Member

May be desirable to extract this method so apps can override to connect to whatever db role. Probably premature.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.