Store newly-uploaded files on save rather than assignment by georgeclaghorn · Pull Request #33303 · rails/rails

georgeclaghorn · 2018-07-06T14:01:02Z

At present, when an uploaded file is assigned to a record via a writer method added by has_one_attached or has_many_attached, it’s persisted to storage immediately. For example, the following causes an uploaded file in params[:avatar] to be stored:

@user.avatar = params[:avatar]

We’d like to support validating blobs attached to records, but for validations to be effective, they ought to prevent invalid files from being stored. It’s of little use to identify an invalid file after it’s already been shipped off to storage: you might use a size validation to limit the cost that a single file can add to your AWS bill, but if the file is stored before validations run, you incur its cost regardless.

This PR changes Active Storage to store attachables assigned to a record after the record is saved rather than immediately. To that end, it introduces change tracking for Active Storage attachments. When you use a writer method added by has_one_attached or has_many_attached, Active Storage tracks a pending change to the record’s attachments. The change is persisted in the application’s database when the record is saved, and the attachables involved are uploaded to storage only after the save transaction commits.

Included are supplementary reorganization and expansion of the attachment API’s tests.

kaspth

Nice! I attempted a cursory review 😄

kaspth · 2018-07-07T15:18:16Z

      CODE

-      has_one :"#{name}_attachment", -> { where(name: name) }, class_name: "ActiveStorage::Attachment", as: :record, inverse_of: :record, dependent: false
+      has_one :"#{name}_attachment", -> { where(name: name) }, class_name: "ActiveStorage::Attachment", as: :record, inverse_of: :record, dependent: :destroy


Does this need a deprecation cycle?

No, it doesn’t change behavior. We did this before by calling purge_later or detach on destroy.

Got it, so it's just more obvious now.

kaspth · 2018-07-07T15:18:52Z

+
+      after_commit(on: [:create, :update]) do
+        begin
+          public_send("#{name}_change").try(:upload)


Why not use &.upload like above?

The Delete* changes don’t respond to upload. They could implement upload and do nothing, and I had them do that originally, but it seemed odd to me.

kaspth · 2018-07-07T15:20:31Z

    # they are all available at the model level.
    def attachment
-      record.public_send("#{name}_attachment")
+      change.present? ? change.attachment : record.public_send("#{name}_attachment")


change&.attachment || record.public_send("#{name}_attachment")?

Tricky: we don’t want to fall back to the attachment in the DB when a Delete* change is pending.

kaspth · 2018-07-07T15:22:56Z

-
-        blob_was.purge_later if blob_was && dependent == :purge_later
+      if !attached? || new_blob != blob
+        write_attachment build_attachment(blob: new_blob)


Should this write a "change" to the accessor? I don't understand how calling attach would now ensure the Attachment got uploaded?

We can explore implementing attach in terms of the writer in the future. I chose to preserve its existing behavior for now. It uses create_blob_from, which uploads as necessary.

Ah, got it 👍 There's a fair amount of different flows here 😄

kaspth · 2018-07-07T15:23:27Z

+
      def build_attachment(blob:)
-        ActiveStorage::Attachment.new(record: record, name: name, blob: blob)
+        Attachment.new(record: record, name: name, blob: blob)


Could also strip ActiveStorage:: from the change classes.

kaspth · 2018-07-07T15:25:57Z

  delegate_missing_to :blob

  after_create_commit :analyze_blob_later, :identify_blob
+  after_destroy_commit :purge_dependent_blob_later


We're switching to delete below because of this, right? Is this callback just to ensure that we always honor the dependent: :destroy from the macros?

Yes. This, along with the switch to dependent: :destroy on attachment associations, keeps changes from having to track removed attachments and purge them on commit: they can just replace the associated attachments and let AR destroy the removed ones.

kaspth · 2018-07-07T15:29:26Z

+        end
+
+        def clear_#{name}_change
+          @active_storage_attached_#{name}_change = nil


It might be nice to replace these ivars with a private/nodoc'ed attachment_changes hash in the vein of Active Model's changes. Then the change classes could use that and skip the need for public_send.

Agree. I'll hook that up.

exviva · 2018-07-13T15:10:00Z

+
+    def upload
+      case attachable
+      when ActionDispatch::Http::UploadedFile, Rack::Test::UploadedFile


Shouldn't we first check if these constants are defined? Are we sure they're guaranteed to be required by now?

Are we sure they're guaranteed to be required by now?

Yes. They’re required in lib/active_storage/attached.rb.

Cool, thanks!

yves-vogl · 2018-07-17T18:38:00Z

Thanks!

tobscher · 2018-09-14T14:39:13Z

@georgeclaghorn That is some excellent work to improve ActiveStorage. It looks like this would potentially fix issue #31985.

If this fixes the issue I mentioned before is there any way to benefit from this fix prior to Rails 6?

In Rails < 6, attachment changes are persisted as soon as they are assigned, regardless of whether the record they’re attached to is successfully saved. That’s why Rails < 6 doesn’t support validation of attachments. We need to, but the model is left in an inconsistent state if validation fails: the old blob is deleted as soon as the new attachment is assigned, but if validation fails the old attachment is preserved, and points to a dead blob. Therefore, we’re monkey patching rails to proactively delete any extant attachments that point to a blob before purging it. This will only be necessary until Rails 6 comes out, because of rails/rails#33303

@georgeclaghorn

In Rails updating an Active Storage relation will now replace the entire association instead of merely adding to it. #35817 (comment) Fixes #35817 cc @georgeclaghorn

glaszig · 2019-05-02T19:18:57Z

    # All methods called on this proxy object that aren't listed here will automatically be delegated to +attachments+.
    def attachments
-      record.public_send("#{name}_attachments")
+      change.present? ? change.attachments : record.public_send("#{name}_attachments")


this either returns an Array or ActiveStorage::Attachment::ActiveRecord_Associations_CollectionProxy which makes the interface quite a moving target. with the latter i can do something like attachments.where(...) whereas that breaks when getting an array.

Store newly-uploaded files on save rather than assignment rails/rails#33303

pjmartorell · 2019-06-04T21:48:23Z

@georgeclaghorn Is there any plan to add this fix/feature to Rails 5.2 as well? it would be really appreciated 🙏

inopinatus · 2019-09-10T22:38:46Z

This PR broke the usual cycle of validation before save.

The justification appears to be cost optimisation for folks like Github and Basecamp.

There is now no official way to access content prior to upload.

I’m now seeing repeated “wtf?”-level questions as a result in a variety of public and private fora.

There’s a job to finish here, because out in meat-and-potatoes business apps land where we just want to validate content before save as normal, programmer happiness has been diminished.

georgeclaghorn · 2019-09-10T22:55:20Z

This PR broke the usual cycle of validation before save.

This PR made it possible to validate file uploads at all. Previously, files were indiscriminately uploaded before validation.

The justification appears to be cost optimisation for folks like Github and Basecamp.

No.

There is now no official way to access content prior to upload.

There wasn’t a way to access pre-upload file data in 5.2, either.

I’m now seeing repeated “wtf?”-level questions as a result in a variety of public and private fora.

There’s a job to finish here, because out in meat-and-potatoes business apps land where we just want to validate content before save as normal, programmer happiness has been diminished.

~~I’m going to take a look eventually, but I’ve been repeatedly annoyed by your tone on here. Adjust it if you’d like to participate further.~~

Edit: I misread Josh’s comment as a demand directed at me. I apologize for reacting so defensively.

inopinatus · 2019-09-11T06:01:48Z

There wasn’t a way to access pre-upload file data in 5.2, either.

Just to be clear, since there might be a terminology misunderstanding here:

By "save", I mean precisely calling #save on a model object, and not any more general notion, and definitely not as as a synonym for uploading a blob; and
By "validate", I mean using Active Model validation callbacks to check the validity of a model's attributes prior to its insertion into the database, and preventing that insertion if validation fails, and including in this the desire to check the validity and acceptability of the content of attachment fields on that model.

In 5.2.3, there was no moment of "pre-upload" at all. As you say, attachments are always uploaded straight away. As a result, blob#download always works if there is an attachment at all, and this validation has been working perfectly well in 5.2.3:

class ImageValidator < ActiveModel::EachValidator
  def validate_each(record, attribute, value)
    unless value.attached?
      record.errors.add attribute, :blank
      return
    end

    if options[:size]
      record.errors.add attribute, :size unless options[:size] === value.byte_size
    end

    if options[:content]
      record.errors.add attribute, :content unless options[:content] === value.download
    end

    record.errors.add attribute, :unrepresentable unless value.representable?
  end
end

class User < ApplicationRecord
  class ContentCheck
    # Dummy content check for demonstration purposes.
    def ===(content)
      Digest::SHA256.hexdigest(content) =~ /\A[a-f]/
    end
  end

  has_one_attached :avatar
  validates :avatar, image: { size: ...1.megabyte, content: ContentCheck.new }

  #...
end

In 6.0.0, this throws an exception in the validator when it hits value.download. The context here is super mundane: submitting a form with a file field to a #create action which does @user = User.new(params) and then if @user.save and so on. As with so many business apps, it really is that dull and ordinary a case.

To put it simply, I am only concerned about having access to the content, whether it is in its final resting place or not, and perceive the loss of that as a regression. It is the model's validation-before-save that matters to us, and not whether or not a loose blob made it to S3 and needs cleaning up later.

I can work around this in a few ways; the simplest, especially for those doing a 5.2 -> 6.0 upgrade, seems to be by using an undocumented internal API, c.f. #37005.

inopinatus · 2019-09-13T00:27:41Z

Edit: I misread Josh’s comment as a demand directed at me. I apologize for reacting so defensively.

George, my remark was evidently ambiguous in that regard and I apologise both for the miscommunication and that it stung you.

jbielick · 2022-02-08T15:24:12Z

It would have been helpful for this information to be in the guides if not the changelog / upgrade guide itself. These changes were breaking and had no deprecation cycle. The work here is beneficial for many I'm sure, but the loop isn't closed on this one.

In my case, validations included analyzing the attached blob. Like other issues linked here, a FileNotFound became the error raising across our application/test suite.

trevorturk · 2022-02-08T18:14:44Z

@jbielick please do consider improving the documentation if you're able! https://guides.rubyonrails.org/v5.1/contributing_to_ruby_on_rails.html#contributing-to-the-rails-documentation

jbielick · 2022-02-08T22:31:42Z

@trevorturk The last PR to add some information about this was closed #37005

I think closing the loop in the sense that I meant it would be providing a public API to retain existing behavior. Is there a workaround you know could be documented?

trevorturk · 2022-02-08T22:39:26Z

No, I was just reacting to you saying "it would have been helpful for this information to be in the guides" and suggesting you improve the documentation if you're able. If you think changes to the code should be made, you should probably consider opening a new issue referencing this one, or ideally a pull request with your proposed changes.

georgeclaghorn force-pushed the activestorage-change-tracking branch from 40c9962 to 5f1d25b Compare July 6, 2018 14:05

georgeclaghorn added the activestorage label Jul 6, 2018

georgeclaghorn force-pushed the activestorage-change-tracking branch 3 times, most recently from b8fbd4c to 37ab8f3 Compare July 6, 2018 22:04

Store newly-uploaded files on save rather than assignment

62856d1

georgeclaghorn force-pushed the activestorage-change-tracking branch from 37ab8f3 to 62856d1 Compare July 6, 2018 22:09

kaspth reviewed Jul 7, 2018

View reviewed changes

georgeclaghorn added 6 commits July 7, 2018 14:20

Track attachment changes centrally

aa09aff

Remove duplicated test

21d4185

attachments -> attached

68e2a45

Add magic comment

7b81dd8

Incorporate attachment reflection fix

268542b

Upload files before app-space after_commit callbacks run

5ff2904

georgeclaghorn force-pushed the activestorage-change-tracking branch from c416740 to 5ff2904 Compare July 8, 2018 02:28

georgeclaghorn merged commit e8682c5 into master Jul 8, 2018

georgeclaghorn deleted the activestorage-change-tracking branch July 8, 2018 03:25

exviva reviewed Jul 13, 2018

View reviewed changes

georgeclaghorn mentioned this pull request Jul 13, 2018

Transaction problem of active storage #32449

Closed

georgeclaghorn mentioned this pull request Aug 14, 2018

Blob does not purge #32902

Closed

tleneveu mentioned this pull request Oct 3, 2018

stop active storage upload when model validation fails or transaction abort igorkasyanchuk/active_storage_validations#7

Closed

abhaynikam referenced this pull request Apr 23, 2019

[ci skip] Active Storage: updating associations replaces

45a6cf3

In Rails updating an Active Storage relation will now replace the entire association instead of merely adding to it. #35817 (comment) Fixes #35817 cc @georgeclaghorn

georgeclaghorn mentioned this pull request May 1, 2019

ActiveStorage attached file not found in after_create #36141

Closed

glaszig reviewed May 2, 2019

View reviewed changes

suketa added a commit to suketa/rails_sandbox that referenced this pull request May 18, 2019

Rails 6.0.0rc1 try015

139b2da

Store newly-uploaded files on save rather than assignment rails/rails#33303

suketa added a commit to suketa/rails_sandbox that referenced this pull request May 24, 2019

Rails 6.0.0rc1 try015

856dff8

Store newly-uploaded files on save rather than assignment rails/rails#33303

kaspth mentioned this pull request May 26, 2019

Unable to analyze blob before saving record #35976

Closed

georgeclaghorn mentioned this pull request Aug 8, 2019

Active Storage Uploads Before Validations #36892

Closed

georgeclaghorn mentioned this pull request Aug 20, 2019

ActiveStorage::FileNotFoundError in before_save #36994

Closed

ahmeij mentioned this pull request Aug 21, 2019

Explain how to access attachments before save #37005

Closed

ErikDeBruijn mentioned this pull request Sep 26, 2019

ActiveStorage file doesn't exist in after_create and after_create_commit method #37304

Closed

ramonpm mentioned this pull request Jan 12, 2021

How to predict attachment's URL in Rails 6+ before saving the record? #41096

Closed

ghiculescu mentioned this pull request Nov 17, 2021

ActiveStorage's attach not saving/uploading the attachment until .save is called #43663

Closed

Sefux mentioned this pull request Oct 3, 2023

Fix before_save behavior on attachment upload a-thousand-channels/ORTE-backend#293

Merged

coderabbitai Bot mentioned this pull request Aug 15, 2026

Fix [vuesim]: Use Tempfile for preview of circuits CircuitVerse/CircuitVerse#7789

Open

8 of 13 tasks

Conversation

georgeclaghorn commented Jul 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kaspth left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yves-vogl commented Jul 17, 2018

Uh oh!

tobscher commented Sep 14, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pjmartorell commented Jun 4, 2019

Uh oh!

inopinatus commented Sep 10, 2019

Uh oh!

georgeclaghorn commented Sep 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

inopinatus commented Sep 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

inopinatus commented Sep 13, 2019

Uh oh!

jbielick commented Feb 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trevorturk commented Feb 8, 2022

Uh oh!

jbielick commented Feb 8, 2022

Uh oh!

trevorturk commented Feb 8, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

georgeclaghorn commented Jul 6, 2018 •

edited

Loading

georgeclaghorn commented Sep 10, 2019 •

edited

Loading

inopinatus commented Sep 11, 2019 •

edited

Loading

jbielick commented Feb 8, 2022 •

edited

Loading