Encoding::UndefinedConversionError when downloading content to a file

[kevinrobayna]

Environment details

• OS: Linux
• Ruby version: 3.2.2
• Gem name and version: google-cloud-storage (>= 1.34)

Steps to reproduce

1. Upload either utf-8 encoded string or utf-8 encoded file to a bucket, this does not happen to all files though is quite random.
2. Download from bucket/key to a utf-8 encoded TempFile
3. BANG!

Code example

    # For context the bucket_store gem it's mentioned in the stacktrace is https://github.com/gocardless/bucket-store/blob/master/lib/bucket_store/gcs.rb I just copied the code here
    @storage = Google::Cloud::Storage.anonymous({timeout: 30})
    
    def get_bucket(name)
      storage.bucket(name, skip_lookup: true)
    end
    
    def upload!(bucket:, key:, file:)
      get_bucket(bucket).create_file(file, key)

      {
        bucket: bucket,
        key: key,
      }
    end

    def download(bucket:, key:, file:)
      file.tap do |f|
        get_bucket(bucket).
          file(key).
          download(f)

        f.rewind
      end
    end

    # It fails
    input_file = Tempfile.new(:encoding => 'utf-8' | default)
=> #<File:/tmp/20240301-21-bzod85>
    download(bucket: "a bucket", key: "a_key", file: input_file)    
    # It works
    input_file = Tempfile.new(:encoding => 'ascii-8bit')
=> #<File:/tmp/20240301-21-bzod85>
    download(bucket: "a bucket", key: "a_key", file: input_file)

Full backtrace

"\xC3" from ASCII-8BIT to UTF-8
/rbenv/versions/3.2.2/lib/ruby/3.2.0/delegate.rb:349:in `write'
/rbenv/versions/3.2.2/lib/ruby/3.2.0/delegate.rb:349:in `block in delegating_block'
/srv/app/vendor/bundle/ruby/3.2.0/gems/google-apis-core-0.14.0/lib/google/apis/core/storage_download.rb:70:in `block in execute_once'
/srv/app/vendor/bundle/ruby/3.2.0/gems/httpclient-2.8.3/lib/httpclient.rb:1093:in `block in follow_redirect'
/srv/app/vendor/bundle/ruby/3.2.0/gems/httpclient-2.8.3/lib/httpclient.rb:1250:in `block in do_get_block'
/srv/app/vendor/bundle/ruby/3.2.0/gems/httpclient-2.8.3/lib/httpclient/session.rb:894:in `read_body_length'
/srv/app/vendor/bundle/ruby/3.2.0/gems/httpclient-2.8.3/lib/httpclient/session.rb:587:in `get_body'
/srv/app/vendor/bundle/ruby/3.2.0/gems/httpclient-2.8.3/lib/httpclient.rb:1247:in `do_get_block'
/srv/app/vendor/bundle/ruby/3.2.0/gems/httpclient-2.8.3/lib/httpclient.rb:1019:in `block in do_request'
/srv/app/vendor/bundle/ruby/3.2.0/gems/httpclient-2.8.3/lib/httpclient.rb:1133:in `protect_keep_alive_disconnected'
/srv/app/vendor/bundle/ruby/3.2.0/gems/httpclient-2.8.3/lib/httpclient.rb:1014:in `do_request'
/srv/app/vendor/bundle/ruby/3.2.0/gems/httpclient-2.8.3/lib/httpclient.rb:1104:in `follow_redirect'
/srv/app/vendor/bundle/ruby/3.2.0/gems/httpclient-2.8.3/lib/httpclient.rb:854:in `request'
/srv/app/vendor/bundle/ruby/3.2.0/gems/httpclient-2.8.3/lib/httpclient.rb:743:in `get'
/srv/app/vendor/bundle/ruby/3.2.0/gems/google-apis-core-0.14.0/lib/google/apis/core/storage_download.rb:50:in `execute_once'
/srv/app/vendor/bundle/ruby/3.2.0/gems/google-apis-core-0.14.0/lib/google/apis/core/http_command.rb:131:in `block (2 levels) in do_retry'
/srv/app/vendor/bundle/ruby/3.2.0/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
/srv/app/vendor/bundle/ruby/3.2.0/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
/srv/app/vendor/bundle/ruby/3.2.0/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
/srv/app/vendor/bundle/ruby/3.2.0/gems/google-apis-core-0.14.0/lib/google/apis/core/http_command.rb:128:in `block in do_retry'
/srv/app/vendor/bundle/ruby/3.2.0/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
/srv/app/vendor/bundle/ruby/3.2.0/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
/srv/app/vendor/bundle/ruby/3.2.0/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
/srv/app/vendor/bundle/ruby/3.2.0/gems/google-apis-core-0.14.0/lib/google/apis/core/http_command.rb:118:in `do_retry'
/srv/app/vendor/bundle/ruby/3.2.0/gems/google-apis-core-0.14.0/lib/google/apis/core/http_command.rb:109:in `execute'
/srv/app/vendor/bundle/ruby/3.2.0/gems/google-apis-core-0.14.0/lib/google/apis/core/base_service.rb:477:in `execute_or_queue_command'
/srv/app/vendor/bundle/ruby/3.2.0/gems/google-apis-storage_v1-0.34.0/lib/google/apis/storage_v1/service.rb:2617:in `get_object'
/srv/app/vendor/bundle/ruby/3.2.0/gems/google-cloud-storage-1.49.0/lib/google/cloud/storage/service.rb:580:in `block in download_file'
/srv/app/vendor/bundle/ruby/3.2.0/gems/google-cloud-storage-1.49.0/lib/google/cloud/storage/service.rb:943:in `execute'
/srv/app/vendor/bundle/ruby/3.2.0/gems/google-cloud-storage-1.49.0/lib/google/cloud/storage/service.rb:579:in `download_file'
/srv/app/vendor/bundle/ruby/3.2.0/gems/google-cloud-storage-1.49.0/lib/google/cloud/storage/file.rb:1046:in `download'
/srv/app/vendor/bundle/ruby/3.2.0/gems/bucket_store-0.6.0/lib/bucket_store/gcs.rb:49:in `block in download'
<internal:kernel>:90:in `tap'
/srv/app/vendor/bundle/ruby/3.2.0/gems/bucket_store-0.6.0/lib/bucket_store/gcs.rb:46:in `download'
/srv/app/vendor/bundle/ruby/3.2.0/gems/bucket_store-0.6.0/lib/bucket_store/key_storage.rb:45:in `download'

[kevinrobayna]

An update on this, when using binmode: true it works and i can download ignoring the encoding