Always save ActiveStorage::Blob before uploading to service

[julik]

Summary

Closes #34805

This PR will be contentious as there is an assumption made that it is possible, with a sufficiently random key, to permit the user to upload to the storage without reserving that key first. I believe the assumption is flawed because it makes the following situation possible:

• User A uploads a PDF of an important legal document (I'm exaggerating, on purpose)
• 4 months pass
• User B edits his comment on something totally unrelated and ponders adding a NSFW picture as an attachment to that comment. Because the user A is unlucky, the picture gets issued the same key as the legal document user A has already uploaded.
• The text editing control user B is using preemptively uploads the file under the key so that saving the comment is fast once the user is done writing. It does receive an error from the ActiveStorage endpoint and tries again. The legal document of the user A is at this point overwritten, and at this stage has been irrecoverably lost. The legal document has been replaced with the NSFW picture.

So I believe, however unlikely (the value spread for the key is very very large indeed) this introduces a possibility of a very sneaky data corruption, which would overwrite data uploaded by the users of the application. This is exacerbated by the fact that when you generate a pre-signed URL to an external service the service will most likely not verify the checksum for you. This means that a person meaning to download the legal document of User A is going to receive the NSFW picture from User B, without any warning.

I know the solution proposed here was probably not the one desired (not holding a DB transaction during upload is a valid requirement) but I believe having risks of silent user data clobbering is worse.

There are a few ways to solve this problem:

• Only use storage backends which support "upload here or error if already taken". Off the top of my head only Redis supports this with SETNX, and then again you need to hold the entire set in memory in Redis for this to work. S3 will happily overwrite on any PUT (it will also overwrite your checksum value)
• Pre-reserve keys using a deterministic sequence number obtained from the database. I tried this and it causes deadlocks on MySQL and is hard to do if your database does not have native cheap sequences. PostgreSQL does but most other databases do not.
• And the last option: insert the row into the database before allowing the upload to take place, so that the database observes the constraint and refuses if the key is taken.

I think part of the reason for this is the method naming for generate_unique_secret_token, because despite it is name it offers no actual guarantees of uniqueness - and it does recommend adding a database constraint on top. The problem here is rather we deliberately allow a state change on an external service which cannot be rolled back before allowing this index to fire.

I've left the test failing for now because they make assumptions about how build... is supposed to work in this case. I do believe that these assumptions should be changed to prevent user data corruption, but would like to poll first.

If the concern is holding a DB transaction open maybe the scope of this transaction should somehow be reduced to key reservation only, I just don't know if ActiveRecord has this facility or not.

Empirical evidence - I work for a company that does lots of file uploads, and I did see wrong things happen, more than once. Some of those wrong things did involve probabilities and preemptive key generation.

[lsylvester]

I think that this makes the documentation for this method ("returns new, unsaved blob instance") incorrect and the method name misleading.

I think that if the possible key collision is an issue, the build_* methods should be removed - and we should instead corresponding create_* methods.

I If we are creating before uploading I don't think that the reserve_key! method would still be required.

[julik]

I think that if the possible key collision is an issue, the build_* methods should be removed - and we should instead corresponding create_* methods.

I would vote for that too. Don't know how much "public-ness" is attached to those build methods though, so not my call to make. "Create before upload always" is the proposed solution anyway.

[georgeclaghorn]

I think I’m okay with removing build_after_upload. build_after_unfurling will likely need to stay to preserve desirable behaviors of the attaching API, but it’s strictly internal and used safely.

[georgeclaghorn]

This method doesn’t need to reserve the resulting blob’s key. In fact, we don’t want it to do so: it’s used to build a blob that is saved with an attached record and uploaded only after the create transaction commits. If that transaction rolls back, the unused blob should not be left behind in the DB.

We can “just” skip reserving the key here since our usage of this method means we’re protected by the uniqueness constraint on the key column.

[georgeclaghorn]

This method’s name no longer matches its behavior. create_and_upload!?

[julik]

@georgeclaghorn I'll try to decipher

This method doesn’t need to reserve the resulting blob’s key. In fact, we don’t want it to do so: it’s used to build a blob that is saved with an attached record and uploaded only after the create transaction commits. If that transaction rolls back, the unused blob should not be left behind in the DB.

Do I understand correctly that in case of build_after_unfurling the actual upload takes place in an after_commit?

I think I’m okay with removing build_after_upload. build_after_unfurling will likely need to stay to preserve desirable behaviors of the attaching API, but it’s strictly internal and used safely.

I don't see it used from anywhere but the Blob class itself, so that's a 👍 from me if agreement has been reached.

[georgeclaghorn]

Do I understand correctly that in case of build_after_unfurling the actual upload takes place in an after_commit?

That’s correct.

[julik]

Ok, so 🔪 build_after_upload is what we go for?

[georgeclaghorn]

👍

[julik]

Will do. What can we do for those still on Rails 5 in this situation?

[georgeclaghorn]

We can backport this without the method removals: add create_and_upload! and prefer it internally.

[julik]

@georgeclaghorn if there is a RecordNotUnique should recovery be attempted? Or do we consider it a "once in a lifetime" event?

[georgeclaghorn]

I’m fine with not recovering.

[julik]

@georgeclaghorn Hopefully all good now. I rebased to take the create blob calls in ActionText tests along.

[julik]

Any chance of this making 6.0?

[julik]

@georgeclaghorn Any chance of getting this to move forward?

[cpsoinos]

Any word on when this will be merged? It's been causing a lot of problems in my app.

[julik]

@cpsoinos on Rails 5 I use this patch for now but indeed would be great if this could get merged (and hopefully even backported). https://gist.github.com/julik/4195b74774dc21dcd2cbd694a25102ce

[cpsoinos]

@julik thanks for the gist! After seeing this PR I implemented a patch myself that essentially runs the logic you have in create_and_upload!, though that's in the business-logic layer of my app. Currently monitoring how it does.

[julik]

@georgeclaghorn is there anything I can do?

[julik]

@georgeclaghorn Would you like me to rebase and update?

[georgeclaghorn]

Hi Julik,

Sorry for the delay here. Please do rebase.

Last thing: on revisiting this, I don’t think we can remove .build_after_upload and .create_after_upload without deprecation. Can you restore them and add deprecation warnings? Then we can remove them in 6.2.

[julik]

Glad we can back to 🔧 on this.

Can I make create_after_upload an alias for create_and_upload + deprecation (which it factually is)?

[georgeclaghorn]

I would personally prefer to restore them with their original behavior, but I don’t feel strongly either way.

[julik]

I opted for aliasing because the original behaviour of create_after_upload leads to the original issue (overwriting), while I believe just having it do something sensible instead and not break API consumers should be good enough ;-)

[georgeclaghorn]

Can you please add a changelog entry?

[georgeclaghorn]

Backported to 6-0-stable (without the deprecations) in 7e23c05.