Unable to use fog-aws with DigitalOcean Spaces: MissingContentLength

[stanhu]

Here is my test script:

require 'rubygems'
require 'fog-aws'

# create a connection
connection = Fog::Storage.new({
  :provider                 => 'AWS',
  :region => 'ams3',
  :aws_access_key_id        => 'my-access-key',
  :aws_secret_access_key    => 'my-secret-key',
  :endpoint => 'https://ams3.digitaloceanspaces.com'
})

directory = connection.directories.get('stanhu-test')

file = directory.files.create(
  :key    => 'unicorn.txt',
  :body   => File.open("/tmp/unicorn.txt"),
  :public => true,
  :length => 45602
)

When I do this, I see:

/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/excon-0.57.1/lib/excon/middlewares/expects.rb:7:in `response_call': Expected(200) <=> Actual(411 Length Required) (Excon::Error::LengthRequired)
excon.error.response
  :body          => "<?xml version=\"1.0\" encoding=\"UTF-8\"?><Error><Code>MissingContentLength</Code><BucketName>stanhu-test</BucketName><RequestId>tx000000000000002f88c3b-005a6055c4-aec4-ams3a</RequestId><HostId>aec4-ams3a-ams3</HostId></Error>"
  :cookies       => [
  ]
  :headers       => {
    "Accept-Ranges"             => "bytes"
    "Content-Length"            => "222"
    "Content-Type"              => "application/xml"
    "Date"                      => "Thu, 18 Jan 2018 08:07:32 GMT"
    "Strict-Transport-Security" => "max-age=15552000; includeSubDomains; preload"
    "x-amz-request-id"          => "tx000000000000002f88c3b-005a6055c4-aec4-ams3a"
  }
  :host          => "ams3.digitaloceanspaces.com"
  :local_address => "10.138.0.7"
  :local_port    => 52494
  :path          => "/stanhu-test/unicorn.txt"
  :port          => 443
  :reason_phrase => "Length Required"
  :remote_ip     => "5.101.110.225"
  :status        => 411
  :status_line   => "HTTP/1.1 411 Length Required\r\n"
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/excon-0.57.1/lib/excon/middlewares/response_parser.rb:9:in `response_call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/excon-0.57.1/lib/excon/connection.rb:388:in `response'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/excon-0.57.1/lib/excon/connection.rb:252:in `request'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/excon-0.57.1/lib/excon/middlewares/idempotent.rb:27:in `error_call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/excon-0.57.1/lib/excon/middlewares/base.rb:11:in `error_call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/excon-0.57.1/lib/excon/middlewares/base.rb:11:in `error_call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/excon-0.57.1/lib/excon/connection.rb:272:in `rescue in request'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/excon-0.57.1/lib/excon/connection.rb:214:in `request'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/excon-0.57.1/lib/excon/middlewares/idempotent.rb:27:in `error_call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/excon-0.57.1/lib/excon/middlewares/base.rb:11:in `error_call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/excon-0.57.1/lib/excon/middlewares/base.rb:11:in `error_call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/excon-0.57.1/lib/excon/connection.rb:272:in `rescue in request'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/excon-0.57.1/lib/excon/connection.rb:214:in `request'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/excon-0.57.1/lib/excon/middlewares/idempotent.rb:27:in `error_call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/excon-0.57.1/lib/excon/middlewares/base.rb:11:in `error_call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/excon-0.57.1/lib/excon/middlewares/base.rb:11:in `error_call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/excon-0.57.1/lib/excon/connection.rb:272:in `rescue in request'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/excon-0.57.1/lib/excon/connection.rb:214:in `request'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/fog-core-1.44.3/lib/fog/core/connection.rb:83:in `request'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/fog-xml-0.1.3/lib/fog/xml/connection.rb:9:in `request'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/fog-aws-1.4.0/lib/fog/aws/storage.rb:613:in `_request'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/fog-aws-1.4.0/lib/fog/aws/storage.rb:607:in `request'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/fog-aws-1.4.0/lib/fog/aws/requests/storage/put_object.rb:48:in `put_object'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/fog-aws-1.4.0/lib/fog/aws/models/storage/file.rb:221:in `save'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/fog-core-1.44.3/lib/fog/core/collection.rb:50:in `create'

I've even tried hacking fog-core to insert the Content-Length header to no avail:

 def request(params, &block)
  params[:headers]['Content-Length'] = 45602    
  puts "requesting #{params[:headers]}"                                                                                                                                                            
  @excon.request(handle_path_prefix_for(params), &block)
end

/cc: @jgannondo

[quai]

@stanhu Your example will work if you pass :body as a string, rather than a File-object. As far as I can see, fog-aws will use streaming / chunked encoding ("x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD") when seeing a File-object, which DigitalOcean might not support?

As a hack in your code, you could check if the endpoint/host matches "digitaloceanspaces.com", and make sure than you pass the backup file as a string, not as a File object?

[stanhu]

@quai Thank you! That was it. I see that there are others who encountered similar issues with other S3 providers:

• https://github.com/minio/minio-boshrelease/issues/5
• Server requires content-length header in chunked encoding (streaming) requests minio/minio#4152

Minio fixed the issue with this comment: minio/minio@b62cd8e

[stanhu]

I now see this comment in fog-aws that leaving out Content-Encoding was intentional as a workaround to solve another problem: 0dee386

[stanhu]

It looks like downgrading aws_signature_version to 2 also works around the problem.

[geemus]

@stanhu glad you found a fix and thanks for sharing.

[jgannondo]

v4 pre-signed URLs should be supported in all of our Spaces regions as of the last ~24 hours, too. If v4 doesn't work for some reason going forward please let me know on this thread, we'd be happy to take a look. Thanks for all the feedback.

[stanhu]

@jgannondo Indeed, it looks like it works now. Thanks!

[geemus]

@jgannondo thanks for the update!

[stanhu]

@jgannondo It looks like uploads via the v4 API work fine, but downloads do not:

When I downgraded to aws_signature_version to 2, the download worked.

[jgannondo]

@stanhu Thanks for letting me know. I forwarded this to the new product manager who is covering Spaces and asked her to respond here. Just wanted to let you know.

[dave08]

Is this problem resolved @jgannondo ? I think I'm still facing it with Gitlab job report downloads (that seem to use fog-aws for this...). I confirm that using the most recent version of Gitlab, and I had to set the signature version to 2 for this to work, so this is still an issue it seems...

https://gitlab.com/gitlab-org/gitlab-ee/issues/10006

[stanhu]

Ok, I just tested:

fog-aws: 2.0.1 uploads ✅, downloads with temp URL: 💥
fog-aws: 3.3.0 uploads ✅, downloads with temp URL: ✅

I believe https://github.com/fog/fog-aws/pull/463/files (#462) solved this issue.

[geemus]

@stanhu great, thanks for updating us!