Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

multipart upload got 477 using aws-sdk-ruby. #490

Merged
merged 4 commits into from Apr 8, 2013

Conversation

Projects
None yet
3 participants
Contributor

ksauzz commented Apr 1, 2013

RiakCS respond 477 when I upload file using multipart upload with aws-sdk-ruby.

to reproduce

git checkout bugfix/kaz-update-ruby-test
make test-ruby

stack trace

/Users/kazuhiro/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:339:in `return_or_raise': AWS::Errors::Base
        from /Users/kazuhiro/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/core/client.rb:440:in `client_request'
        from (eval):3:in `complete_multipart_upload'
        from /Users/kazuhiro/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/s3/multipart_upload.rb:244:in `complete'
        from /Users/kazuhiro/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/s3/multipart_upload.rb:267:in `close'
        from /Users/kazuhiro/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/s3/s3_object.rb:713:in `multipart_upload'
        from /Users/kazuhiro/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/s3/s3_object.rb:1629:in `write_with_multipart'
        from /Users/kazuhiro/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/aws-sdk-1.8.5/lib/aws/s3/s3_object.rb:604:in `write'
        from hoge.rb:16:in `<main>'

code

!#/usr/bin/env ruby

require 'aws-sdk'

s3client = AWS::S3.new(
  access_key_id:  "your-id",
  secret_access_key: "your-secret-key",
  proxy_uri: "http://localhost:8080",
  use_ssl: false,
  max_retries: 0,
)
bucket = s3client.buckets.create 'simple-bucket-xxxxxxxxxxxxxxxxxxxxxxxx'
bucket.objects['multi'].write(:file => 'Rakefile', :multipart_threshold => 100)
puts bucket.objects['multi'].read
Contributor

kellymclaughlin commented Mar 14, 2013

Is it possible the body being sent to finalize the upload is not valid xml? That looks like the only reason a 477 would be returned. Can you try to see what is actually being received as the body of a failing request like that?

Contributor

slfritchie commented Mar 15, 2013

Hi, Kaz. The 477 code is a leftover from my debugging work, I'm almost certain.

If the XML is not parseable, or if the uploadId cannot be Base64 decoded, then my WM debugging code returns {halt, 477} to Webmachine. I should've fixed that before a final commit & merge.`

https://github.com/basho/riak_cs/blob/release/1.3/src/riak_cs_wm_object_upload_part.erl#L124

... Oh, the current code is broken: if you submit good XML but an undecodable uploadId, then a bad UploadId value will be passed to riak_cs_mp_utils:complete_multipart_upload(). It isn't clear to me how complete_multipart_upload() will react if given an crash message from the catch.

So, there are 2 bugs here:

  1. parse_body() -> bad should return: HTTP code _____? And return the ____? appropriate XML error stuff in the HTTP body for S3 API consistency, e.g. "BadRequest".
  2. decode() -> exception should return: HTTP code _? And return the __? appropriate XML error stuff in the HTTP body for S3 API consistency, e.g. "BadRequest".

Do you happen to know what the proper S3 API codes are for bad XML and for bad uploadId?

Contributor

ksauzz commented Mar 15, 2013

Thank you for the comments.
I found a strange quoted Etag element in a request body by aws-sdk-ruby. :( I guess CS can't parse this.

<CompleteMultipartUpload><Part><PartNumber>1</PartNumber><ETag>&quot;d8d4c2d1c8e4b04d96bca23175d071c5&quot;</ETag></Part></CompleteMultipartUpload>

Related tcpdumps are follows:

RiakCS with aws-sdk-ruby

POST http://aws-sdk-test-6053dfa0-6f52-0130-3371-7cd1c3e4679b.s3.amazonaws.com/key-610e1900-6f52-0130-3371-7cd1c3e4679b?uploadId=beJIU-9yQOytlJeA-wt3YQ%3D%3D HTTP/1.1
Content-Type: 
Content-Length: 147
User-Agent: aws-sdk-ruby/1.7.1 ruby/1.9.3 x86_64-darwin12.0.0
Date: Fri, 15 Mar 2013 12:57:28 +0900
Authorization: AWS OMJTFL1EFU8FBEVEOV0Y:3kgEhV2nWnBrfbwYwTM2Dvigg8E=
Accept: */*
Host: aws-sdk-test-6053dfa0-6f52-0130-3371-7cd1c3e4679b.s3.amazonaws.com

<CompleteMultipartUpload><Part><PartNumber>1</PartNumber><ETag>&quot;d8d4c2d1c8e4b04d96bca23175d071c5&quot;</ETag></Part></CompleteMultipartUpload>HTTP/1.1 477 Internal Server Error
Server: Riak CS
Date: Fri, 15 Mar 2013 03:57:28 GMT
Content-Type: application/xml
Content-Length: 0

RiakCS with s3cmd

POST http://hogehogehoge.s3.amazonaws.com/riak-1.2.1-b416b631.tar.gz?uploadId=eWgMGxZjTReT_ISopX17WQ== HTTP/1.1
Host: hogehogehoge.s3.amazonaws.com
Accept-Encoding: identity
content-length: 223
Authorization: AWS -NUGH32ULSAASTCZFR02:6c1VDoXo9CX8Mf5w/+kPT3nWZYM=
x-amz-date: Fri, 15 Mar 2013 03:54:12 +0000

<CompleteMultipartUpload><Part><PartNumber>1</PartNumber><ETag>"963e93ec3223e680085088237b55a384"</ETag></Part><Part><PartNumber>2</PartNumber><ETag>"16e221152cc4bdc18d8c80f9f3a695d4"</ETag></Part></CompleteMultipartUpload>HTTP/1.1 200 OK
Server: Riak CS
Date: Fri, 15 Mar 2013 03:54:12 GMT
Content-Type: application/xml
Content-Length: 336

<?xml version="1.0" encoding="UTF-8"?><CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Location>http://hogehogehoge.s3.amazonaws.com/riak-1.2.1-b416b631.tar.gz</Location><Bucket>hogehogehoge</Bucket><Key>riak-1.2.1-b416b631.tar.gz</Key><ETag>eWgMGxZjTReT_ISopX17WQ==</ETag></CompleteMultipartUploadResult>

AWS with aws-sdk-ruby

POST /multi?uploadId=NWbA_LwxGcXkMfsDxP4mkrjlQFnC5nidhVRSoQfzq09aFb_aFl9GI72PDXSeGaEQXx6SsumU5P3mXdA9dZa0fA-- HTTP/1.1
Content-Type: 
Content-Length: 147
User-Agent: aws-sdk-ruby/1.8.5 ruby/1.9.3 x86_64-darwin12.0.0
Date: Fri, 15 Mar 2013 04:59:14 GMT
Authorization: AWS AKIAJBZS4IGVQGRQBHVA:M8qyhpg1VPJSQ4uvxQeY/WgPgew=
Accept: */*
Host: simple-bucket-xxxxxxxxxxxxxxxxxxxxxxxx.s3.amazonaws.com

<CompleteMultipartUpload><Part><PartNumber>1</PartNumber><ETag>&quot;32ae611e9cd03990d2933c89396b3ee2&quot;</ETag></Part></CompleteMultipartUpload>HTTP/1.1 200 OK
x-amz-id-2: q+pbw5r085WOjD58P8f1NsBSHce0hS1Odot7uTmrW3LhN7karjMv//yO5TSiP+gQ
x-amz-request-id: 88ED8909A725DAA8
Date: Fri, 15 Mar 2013 04:59:12 GMT
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3

27
<?xml version="1.0" encoding="UTF-8"?>

14b

<CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Location>http://simple-bucket-xxxxxxxxxxxxxxxxxxxxxxxx.s3.amazonaws.com/multi</Location><Bucket>simple-bucket-xxxxxxxxxxxxxxxxxxxxxxxx</Bucket><Key>multi</Key><ETag>&quot;565f0e5818ad495d75a297666e48c33f-1&quot;</ETag></CompleteMultipartUploadResult>
0

@ghost ghost assigned ksauzz Mar 15, 2013

@kellymclaughlin kellymclaughlin commented on the diff Apr 2, 2013

@@ -57,6 +57,10 @@ test-client: test-clojure test-python test-erlang
test-python: test-boto
+test-ruby:
@kellymclaughlin

kellymclaughlin Apr 2, 2013

Contributor

Any reason not to also add test-ruby to the test-client target so it runs with make test-client also?

@ghost ghost assigned kellymclaughlin Apr 2, 2013

@kellymclaughlin kellymclaughlin commented on an outdated diff Apr 2, 2013

client_tests/ruby/spec/helper.rb
+ response = HTTParty.put("http://localhost:8080/riak-cs/user",
+ :body => {
+ :name => name,
+ :email => "#{name}@example.com"}.to_json,
+ :headers => {
+ "Content-Type" => "application/json"})
+ json_body = JSON.parse(response.body)
+ return json_body['key_id'], json_body['key_secret']
+end
+
+def s3_conf
+ key_id, key_secret = create_user(UUID::generate)
+ {
+ access_key_id: key_id,
+ secret_access_key: key_secret,
+ proxy_uri: "http://localhost:8080",
@kellymclaughlin

kellymclaughlin Apr 2, 2013

Contributor

Could you change this in a similar manner to the python test module so it could also be run with riak_test?

Contributor

ksauzz commented Apr 3, 2013

Thanks. I'll do them.

Contributor

kellymclaughlin commented Apr 5, 2013

Looks good. +1 to merge. After merging to release/1.3 we should probably go ahead and merge release/1.3 to develop again so we can have this test run as part of new development sooner rather than later.

@ghost ghost assigned ksauzz Apr 5, 2013

ksauzz added a commit that referenced this pull request Apr 8, 2013

Merge pull request #490 from basho/bugfix/kaz-update-ruby-test
multipart upload got 477 using aws-sdk-ruby.

@ksauzz ksauzz merged commit e0aabc5 into release/1.3 Apr 8, 2013

@ksauzz ksauzz deleted the bugfix/kaz-update-ruby-test branch Apr 8, 2013

Contributor

ksauzz commented Apr 8, 2013

Sure. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment