MD5 verification error when downloading a specific file generation

[ozziek]

Environment details

• OS: MacOS
• Ruby version: ruby 2.5.5p157 (2019-03-15 revision 67260) [x86_64-darwin18]
• Gem name and version: google-cloud-storage (1.19.0)

Steps to reproduce

• Enable bucket versioning
• Upload a .json file
• Get the generation number for it
• Upload a new .json which replaces the previous file
• Attempt to download the previous file

Code example

data = JSON.pretty_generate({ "name" => "Roger" })
# => "{\n  \"name\": \"Roger\"\n}"

# generate brand new file
bucket.create_file(StringIO.new(data), "test_1.json")
=> #<Google::Cloud::Storage::File: ... >

prev_generation = bucket.file("test_1.json").generation
# => 1564749365636895

# update the data with different infromation
data = JSON.pretty_generate({ "name" => "Ozzie" })
# => "{\n  \"name\": \"Ozzie\"\n}"

# upload to the same location, forcing a new version
bucket.create_file(StringIO.new(data), "test_1.json")
# => #<Google::Cloud::Storage::File: ... >

# parsing from the current file works fine
current_file = bucket.file("test_1.json").download
JSON.parse(current_file.read) 
# => {"name"=>"Ozzie"}

# parsing the previous file does not
prev_file = bucket.file("test_1.json", generation: prev_generation).download

# Google::Cloud::Storage::FileVerificationError: The downloaded file failed MD5 verification.
# from ruby/2.5.5/gems/google-cloud-storage-1.19.0/lib/google/cloud/storage/file/verifier.rb:34:in `verify_md5!'

Full backtrace

Google::Cloud::Storage::FileVerificationError: The downloaded file failed MD5 verification.
from /Users/ozziekirkbu/.gem/ruby/2.5.5/gems/google-cloud-storage-1.19.0/lib/google/cloud/storage/file/verifier.rb:34:in `verify_md5!'

Notes

• Copying the older generation file from gsutil works fine.
• Might be related to the download method. It doesn't seem to include the generation number when downloading the file.

[quartzmo]

@kirkbyo Thanks for reporting this issue. I'll try to reproduce it today.

[ozziek]

@quartzmo Great thank you! Let me know if you need any other additional information.

[quartzmo]

I've reproduced this in a new acceptance test:

  1) Error:
Google::Cloud::Storage::File::storage#test_0002_should upload, replace and download a previous generation of a file:
Google::Cloud::Storage::FileVerificationError: The downloaded file failed MD5 verification.
    /Users/quartzmo/code/google/codez/google-cloud-ruby/google-cloud-storage/lib/google/cloud/storage/file/verifier.rb:34:in `verify_md5!'
    /Users/quartzmo/code/google/codez/google-cloud-ruby/google-cloud-storage/lib/google/cloud/storage/file.rb:1763:in `verify_file!'
    /Users/quartzmo/code/google/codez/google-cloud-ruby/google-cloud-storage/lib/google/cloud/storage/file.rb:945:in `download'
    /Users/quartzmo/code/google/codez/google-cloud-ruby/google-cloud-storage/acceptance/storage/file_test.rb:96:in `block (3 levels) in <top (required)>'
    /Users/quartzmo/.rbenv/versions/2.4.5/lib/ruby/2.4.0/tempfile.rb:295:in `open'
    /Users/quartzmo/code/google/codez/google-cloud-ruby/google-cloud-storage/acceptance/storage/file_test.rb:95:in `block (2 levels) in <top (required)>'

[quartzmo]

@kirkbyo The fix for this issue was just released in google-cloud-storage/versions/1.20.0.