feat(storage): Add full object checksum validation for appendable uploads finalize#16245
Conversation
There was a problem hiding this comment.
Code Review
This pull request refactors checksum handling and hash function management during appendable uploads. The review feedback identifies critical issues where removing the Crc32cHashFunction constructor and its recreation logic breaks checksum validation for resumed non-appendable uploads. Additionally, the reviewer recommends simplifying the UpdateChecksums signature, returning an explicit error if requested CRC32C validation cannot be performed due to missing metadata, and adding unit tests to cover the new validation paths.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #16245 +/- ##
========================================
Coverage 92.26% 92.27%
========================================
Files 2214 2214
Lines 206307 206505 +198
========================================
+ Hits 190355 190546 +191
- Misses 15952 15959 +7 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
a354dfa to
04ce29b
Compare
04ce29b to
a77ed47
Compare
a77ed47 to
69c280b
Compare
69c280b to
76976b5
Compare
#GeminiAlert
is_append == true
For appendable uploads, the internal HashFunction is broken. It only sees the chunks uploaded in the current stream, not the full object, so its locally computed hashes are wrong. Because it's broken, we MUST bypass it (by setting action = kFinalize) and manually inject the checksum ourselves. Since we are doing it manually, we want to respect the user's expected checksum regardless of whether they set it at the start of the upload (options_) OR dynamically at the end (current_options). Therefore, we pull the value from merged to cover both bases.
is_append == false (Normal Uploads)
For normal uploads, the internal HashFunction works perfectly! It computes the hashes over the whole object, validates them locally, and injects them into the gRPC request. HOWEVER, the HashFunction is initialized at the start of the upload. It only knows about options_. It has absolutely zero knowledge of current_options (options passed dynamically at the exact moment of Finalize()).
If the user provided the checksum at the start (options_), we WANT the HashFunction to handle it. If we checked merged here, we would mistakenly bypass the HashFunction. BUT, if the user provided the checksum dynamically at the end (current_options), the HashFunction doesn't know about it. It will compute its own hash and aggressively overwrite whatever the user wanted! Therefore, for normal uploads, we only check current_options. If the user provided a last-second checksum, we bypass the HashFunction and manually inject it. Otherwise, we leave the HashFunction alone to do its job.
Summary:
We use merged for appends because we must manually inject the checksum no matter when it was provided.
We use current_options for normal uploads because we only want to manually inject it if the user provided a last-second checksum that the HashFunction doesn't know about.