Add ActiveStorage::Blob.compose

[gmcgibbon]

Summary

Retry of #37314

I'd like to be able to use GCS' compose feature within Active Storage. Essentially, it allows you to combine multiple files together. This is particularly useful when handling multiple large files.

One issue I can see is that it is impossible to compute the checksum for a file that's assembled on the storage provider's end. In this implementation, composed blobs cannot be opened as they would fail integrity checks. We could make a special blob type ( eg. conposed: true) that skips checksum verification, but I'm not sure if that's a preferred approach for this use-case. What do you think?

[sedubois]

You have my vote of support. I would love to combine files for other reasons than just them being large. For example combining various inputs with vips, ffmpeg, etc (see #39283 to support non-image variants).

[byroot]

particularly useful when handling multiple large files.

One issue I can see is that it is impossible to compute the checksum for a file that's assembled on the storage provider's end.

In the case of GCS at least, you'll get an "object" response which include the MD5, CRC etc.

I know very little of Active Storage, but the compose API could be required to return the composed object checksum. No?

[rails-bot]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Thank you for your contributions.

[zzak]

@gmcgibbon Do you have time to rebase this and address Jean's comments above? 🙏

[rails-bot]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Thank you for your contributions.

[gmcgibbon]

In the case of GCS at least, you'll get an "object" response which include the MD5, CRC etc.

I know very little of Active Storage, but the compose API could be required to return the composed object checksum. No?

Right. I remember now that I tried this but composite objects on GCS don't have MD5 hashes, only CRC32C. Other storage services seem to also not include MD5 hashes. This leaves me no choice but to stream the file locally and calculate the MD5 myself. Though, there may be a more clever way of doing that.

[gmcgibbon]

This seems to work well across services without having to calculate checksums on large files. It may take a slightly assertive approach to changing library expectations in terms of checksums, but composed blobs may be able to depend on the integrity of the individual blobs you are composed of. At least, storage services seem to think that way.