AWS.S3.complete_multipart_upload(client, bucket, key, %{}) returns an error #84

gabrielmancini · 2021-07-26T22:00:10Z

Hello Folk's

Nice lib! :-)

i had an error when try the s3 partial_upload part

the code:

# ...
{:ok, %{
   "InitiateMultipartUploadResult" => %{
     "UploadId" => key
    }}, _} = AWS.S3.create_multipart_upload(client, bucket, path, %{})

    Stream.concat([head], rest)
      |> Stream.chunk_every(chunk_size)
      |> Enum.map(fn chunk ->
      chunk_s =
        chunk
        |> Enum.join("\r\n")
      size = :erlang.byte_size(chunk_s)
      IO.inspect( size / @megabyte)
      AWS.S3.upload_part(client, bucket, key, %{"Body" => chunk_s, "ContentMD5" => :crypto.hash(:md5, chunk_s) |> Base.encode64()})
      |> IO.inspect()
    end)

    AWS.S3.complete_multipart_upload(client, bucket, key, %{})

error

{:error,
 {:unexpected_response,
  %{
    body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>MethodNotAllowed</Code><Message>The specified method is not allowed against this resource.</Message><Method>POST</Method><ResourceType>OBJECT</ResourceType><RequestId>EDCQXBK7XAH5FRKH</RequestId><HostId>bD0jm08lboSt4yjEOsNxXTrxGb/i3rBdNATN+uD+mgTZ+F0w64aFddXj6Vthcrc4ClD8DBX0buo=</HostId></Error>",
    headers: [
      {"x-amz-request-id", "EDCQXBK7XAH5FRKH"},
      {"x-amz-id-2",
       "bD0jm08lboSt4yjEOsNxXTrxGb/i3rBdNATN+uD+mgTZ+F0w64aFddXj6Vthcrc4ClD8DBX0buo="},
      {"allow", "HEAD, DELETE, GET, PUT"},
      {"content-type", "application/xml"},
      {"transfer-encoding", "chunked"},
      {"date", "Mon, 26 Jul 2021 21:52:17 GMT"},
      {"server", "AmazonS3"}
    ],
    status_code: 405
  }}}
** (Protocol.UndefinedError) protocol Enumerable not implemented for {:error, {:unexpected_response, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>MethodNotAllowed</Code><Message>The specified method is not allowed against this resource.</Message><Method>POST</Method><ResourceType>OBJECT</ResourceType><RequestId>EDCQXBK7XAH5FRKH</RequestId><HostId>bD0jm08lboSt4yjEOsNxXTrxGb/i3rBdNATN+uD+mgTZ+F0w64aFddXj6Vthcrc4ClD8DBX0buo=</HostId></Error>", headers: [{"x-amz-request-id", "EDCQXBK7XAH5FRKH"}, {"x-amz-id-2", "bD0jm08lboSt4yjEOsNxXTrxGb/i3rBdNATN+uD+mgTZ+F0w64aFddXj6Vthcrc4ClD8DBX0buo="}, {"allow", "HEAD, DELETE, GET, PUT"}, {"content-type", "application/xml"}, {"transfer-encoding", "chunked"}, {"date", "Mon, 26 Jul 2021 21:52:17 GMT"}, {"server", "AmazonS3"}], status_code: 405}}} of type Tuple. This protocol is implemented for the following type(s): Function, MapSet, List, Stream, HashDict, GenEvent.Stream, Map, Date.Range, Range, File.Stream, IO.Stream, HashSet
    (elixir 1.12.0) lib/enum.ex:1: Enumerable.impl_for!/1
    (elixir 1.12.0) lib/enum.ex:141: Enumerable.reduce/3
    (elixir 1.12.0) lib/stream.ex:649: Stream.run/1

some gotchas here?

The text was updated successfully, but these errors were encountered:

philss · 2021-07-27T21:20:30Z

Hi @gabrielmancini 👋

Just to confirm: is the error occurring inside the stream?

Also, have you tried to use the PartNumber parameter for the upload_part/4 call?

Part numbers can be any number from 1 to 10,000, inclusive. A part number uniquely identifies a part and also defines its position within the object being created. If you upload a new part using the same part number that was used with a previous part, the previously uploaded part is overwritten. Each part must be at least 5 MB in size, except the last part. There is no size limit on the last part of your multipart upload.

https://hexdocs.pm/aws/AWS.S3.html#upload_part/5

gabrielmancini · 2021-08-02T15:52:07Z

hello @philss thanks for the reply ;-)

i change a little bit the code but i had the same issue, pls take a look.

the code:

{:ok,
     %{
       "InitiateMultipartUploadResult" => %{
         "UploadId" => key
       }
     }, _} = AWS.S3.create_multipart_upload(client, bucket, path, %{})

    # the magic "must" happend! "bias detected"
    Stream.concat([head], rest)
    |> Stream.chunk_every(chunk_size)
    |> Stream.with_index(1)
    |> Enum.map(fn {chunk, i} ->
      chunk_s =
        chunk
        |> Enum.join("\r\n")

      size = :erlang.byte_size(chunk_s)
      IO.inspect({i, size / @megabyte})
      AWS.S3.upload_part(client, bucket, path, %{
        "Body" => chunk_s,
        "ContentMD5" => :crypto.hash(:md5, chunk_s) |> Base.encode64(),
        "PartNumber" => i,
        "UploadId" => key
      })
      # every post returns 200 ok
      |> IO.inspect()
    end)

    # https://github.com/aws-beam/aws-elixir/issues/84
    AWS.S3.complete_multipart_upload(client, bucket, path, %{})
    |> IO.inspect()

the error:

{:error,
 {:unexpected_response,
  %{
    body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>MethodNotAllowed</Code><Message>The specified method is not allowed against this resource.</Message><Method>POST</Method><ResourceType>OBJECT</ResourceType><RequestId>RBWVRAPGGPPF746S</RequestId><HostId>z6b1vYUzI5NAdvm/yIzpJ0kt+0iKQ0ZSQAHRpzMgs9BJDZ2Kz0QJ9FYjYpzpO3Ko1J498e+JKgU=</HostId></Error>",
    headers: [
      {"x-amz-request-id", "RBWVRAPGGPPF746S"},
      {"x-amz-id-2",
       "z6b1vYUzI5NAdvm/yIzpJ0kt+0iKQ0ZSQAHRpzMgs9BJDZ2Kz0QJ9FYjYpzpO3Ko1J498e+JKgU="},
      {"allow", "HEAD, DELETE, GET, PUT"},
      {"content-type", "application/xml"},
      {"transfer-encoding", "chunked"},
      {"date", "Mon, 02 Aug 2021 15:48:06 GMT"},
      {"server", "AmazonS3"}
    ],
    status_code: 405
  }}}
** (Protocol.UndefinedError) protocol Enumerable not implemented for {:error, {:unexpected_response, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>MethodNotAllowed</Code><Message>The specified method is not allowed against this resource.</Message><Method>POST</Method><ResourceType>OBJECT</ResourceType><RequestId>RBWVRAPGGPPF746S</RequestId><HostId>z6b1vYUzI5NAdvm/yIzpJ0kt+0iKQ0ZSQAHRpzMgs9BJDZ2Kz0QJ9FYjYpzpO3Ko1J498e+JKgU=</HostId></Error>", headers: [{"x-amz-request-id", "RBWVRAPGGPPF746S"}, {"x-amz-id-2", "z6b1vYUzI5NAdvm/yIzpJ0kt+0iKQ0ZSQAHRpzMgs9BJDZ2Kz0QJ9FYjYpzpO3Ko1J498e+JKgU="}, {"allow", "HEAD, DELETE, GET, PUT"}, {"content-type", "application/xml"}, {"transfer-encoding", "chunked"}, {"date", "Mon, 02 Aug 2021 15:48:06 GMT"}, {"server", "AmazonS3"}], status_code: 405}}} of type Tuple. This protocol is implemented for the following type(s): Function, MapSet, List, Stream, HashDict, GenEvent.Stream, Map, Date.Range, Range, File.Stream, IO.Stream, HashSet
    (elixir 1.12.0) lib/enum.ex:1: Enumerable.impl_for!/1
    (elixir 1.12.0) lib/enum.ex:141: Enumerable.reduce/3
    (elixir 1.12.0) lib/stream.ex:649: Stream.run/1

all the parts appers to work fine, all of then returns status code 200, like this output sample:

{:ok, nil,
 %{
   body: "",
   headers: [
     {"x-amz-id-2",
      "6LK+sRvE0+hmCpMXepbMJIwN6SLhDntv50CKUNcj/fWqGvvBz38oD2kUhIYvgqs/6ZEw6DKEDNg="},
     {"x-amz-request-id", "RBWYDXCATZ5XVHKC"},
     {"date", "Mon, 02 Aug 2021 15:48:08 GMT"},
     {"etag", "\"94efc8234db38870f667962c3425f5a6\""},
     {"server", "AmazonS3"},
     {"content-length", "0"}
   ],
   status_code: 200
 }}

dmorn · 2022-03-24T13:42:42Z

Hi there @gabrielmancini and @philss!
I'm currently facing the same issue. One thing that is missing from above is a proper input payload.
I am building it like this

parts =
  Enum.map(uploads, fn {{:ok, etag}, index} ->
    %{"ETag" => etag, "PartNumber" => index}
  end)

input = %{"CompleteMultipartUpload" => parts, "uploadId" => upload_id}
AWS.S3.complete_multipart_upload(client, bucket, key, input)

The error is the same though

"%{body: \"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?>\\n<Error><Code>MethodNotAllowed</Code><Message>The specified method is not allowed against this resource.</Message><Method>POST</Method><ResourceType>OBJECT</ResourceType><RequestId>C31JVFSFYW0HXKPK</RequestId><HostId>n+/zF/4mGwuKv52tlHEeJbuV1tEn4C8TNhv9T4WXuxTPDecSzxicwKcI8kIngE7bVzgzjB0SbPs=</HostId></Error>\", headers: [{\"x-amz-request-id\", \"C31JVFSFYW0HXKPK\"}, {\"x-amz-id-2\", \"n+/zF/4mGwuKv52tlHEeJbuV1tEn4C8TNhv9T4WXuxTPDecSzxicwKcI8kIngE7bVzgzjB0SbPs=\"}, {\"Allow\", \"HEAD, DELETE, GET, PUT\"}, {\"Content-Type\", \"application/xml\"}, {\"Transfer-Encoding\", \"chunked\"}, {\"Date\", \"Thu, 24 Mar 2022 13:36:15 GMT\"}, {\"Server\", \"AmazonS3\"}, {\"Connection\", \"close\"}], status_code: 405}"

It looks like this endpoint does not allow POST requests (see the Allow response header) 🤔

dmorn · 2022-03-24T13:49:58Z

This is indeed the culprit! Changing

aws-elixir/lib/aws/generated/s3.ex

Line 212 in a80272a

:post,

to a :put makes it work out 😅 On AWS documentation though they actually say this is supposed to be a POST.

dmorn · 2022-03-24T13:55:43Z

It is indeed specified as a POST in the spec document https://github.com/aws/aws-sdk-go/blob/a893727e4b5ed70693aa65bd1b081d4cdcd18a4b/models/apis/s3/2006-03-01/api-2.json#L33

onno-vos-dev · 2022-03-24T13:56:18Z

👋 Not super familiar with the Elixir code but in aws-erlang this is a post and I know for a fact that this seems to work just fine judging by our S3 bucket containing complete gigantic files 🤔

So before going ahead and changing it, I'd like you (or someone else) to really scuba dive and ensure the bug isn't hiding elsewhere.

dmorn · 2022-03-24T13:58:43Z

Hi @onno-vos-dev, thanks for joining in. I do think that this stinks.

onno-vos-dev · 2022-03-24T14:02:45Z

@dmorn Oh I'm not arguing with you 😄 For inspiration, this is how our Input looks which judging by yours appears to be slightly different 🤔 Unless I'm more rusty in my Elixir than I think I am 🤔

Input =
    #{<<"UploadId">> => UploadId,
      <<"CompleteMultipartUpload">> =>
        #{<<"Part">> =>
            [#{<<"ETag">> => Etag, <<"PartNumber">> => PartNr} || {PartNr, Etag} <- PartEtags]}},

dmorn · 2022-03-24T14:17:21Z

Uh uh this is different indeed 😜 My erlang may be rusty now, it this what you're producing here?

%{
  "CompleteMultipartUpload" => [
    %{
      "Part" => %{
        "ETag" => "5592cc66124693a066c16198bc40e330",
        "PartNumber" => 1
      }
    },
    %{
      "Part" => %{
        "ETag" => "75afa18db5dfe9f747e39f03c2152c80",
        "PartNumber" => 2
      }
    },
    %{"Part" => %{"ETag" => "1668a6efbd8003e55abef8d7a48e2851", ...}},
    %{"Part" => %{...}},
    %{...},
    ...
  ],
  "uploadId" => "elided"
}

The uploadId is lowercased here cause it is turned into a query parameter and should not be part of the payload (I double checked both the code and tried it with the uppercased version, I get a malformed input error from AWS)

dmorn · 2022-03-24T14:19:30Z

Anyway it does not matter if I build it like above or like

%{
  "CompleteMultipartUpload" => %{
    "Part" => [
      %{"ETag" => "5592cc66124693a066c16198bc40e330", "PartNumber" => 1},
      %{"ETag" => "75afa18db5dfe9f747e39f03c2152c80", "PartNumber" => 2},
      %{"ETag" => "1668a6efbd8003e55abef8d7a48e2851", ...},
      %{...},
      ...
    ]
  },
  "uploadId" => "VUxIR9hGxsV0i90Gy8FKPI7lNuSyYWpEVVkbYneA65sRDCfL21iDJkEXgDLWJiTHSv8rGaxVmVivAmMpp09xAQ--"
}

The error persists 😅

dmorn · 2022-03-24T14:21:43Z

I think your code produces the second payload I posted @onno-vos-dev right?

onno-vos-dev · 2022-03-24T14:23:49Z

Anyway it does not matter if I build it like above or like

%{
  "CompleteMultipartUpload" => %{
    "Part" => [
      %{"ETag" => "5592cc66124693a066c16198bc40e330", "PartNumber" => 1},
      %{"ETag" => "75afa18db5dfe9f747e39f03c2152c80", "PartNumber" => 2},
      %{"ETag" => "1668a6efbd8003e55abef8d7a48e2851", ...},
      %{...},
      ...
    ]
  },
  "uploadId" => "VUxIR9hGxsV0i90Gy8FKPI7lNuSyYWpEVVkbYneA65sRDCfL21iDJkEXgDLWJiTHSv8rGaxVmVivAmMpp09xAQ--"
}

The error persists sweat_smile

That looks similar to what I generate:

#{<<"CompleteMultipartUpload">> =>
      #{<<"Part">> =>
            [#{<<"ETag">> => <<"etag1">>,<<"PartNumber">> => 1},
             #{<<"ETag">> => <<"etag2">>,<<"PartNumber">> => 2},
             #{<<"ETag">> => <<"etag3">>,<<"PartNumber">> => 3}]},
  <<"UploadId">> => <<"my-elixir-is-terrible">>}

So the plot thickens 🤔 I need to do some thinking and digging.

dmorn · 2022-03-24T14:29:47Z

I just double checked how ExAws works as we do multi-part uploads with that lib in another project. The input payload they produce resembles the first one of the last above and they do indeed POST.

dmorn · 2022-03-24T15:00:17Z

OOOOk fot it @onno-vos-dev. This is all about the input payload as expected. After inspecting the encoded request body a found that a proper elixir input is built like

parts =
  Enum.map(uploads, fn {{:ok, etag}, index} ->
    %{"ETag" => etag, "PartNumber" => index}
  end)

input = %{"CompleteMultipartUpload" => %{"Part" => parts}, "UploadId" => upload_id}

And should produce a payload that looks like

<CompleteMultipartUpload>
   <Part>
      <ETag>a2b0962b15f6d5716b3e9df437a8133b</ETag>
      <PartNumber>1</PartNumber>
   </Part>
   <Part>
   ...
   </Part>
</CompleteMultipartUpload>

So that was it, bad input @gabrielmancini ! I believe this issue can be closed @philss, thank you again @onno-vos-dev! 🙌

onno-vos-dev · 2022-03-24T15:12:03Z

Now I can put Elixir on my CV 😅 Glad to see this is resolved!

Ok if I close this issue?

dmorn · 2022-03-24T15:14:34Z

@onno-vos-dev 😂 Yes I'm OK with it!

ferd · 2024-04-06T17:22:34Z

This is an old issue, but I wanted to leave a code sample for multipart uploading with checksums and hashes included, in case anyone else ever needs it.

As it turns out, you need to upload the checksums of each part you upload, store the ETags given, and then re-send all the sections in the completion step, and the checsum you get back is the checksum of all checksums with a suffix that contains the number of parts involved; any moving of that file after the fact (if under 5GB) re-writes the checksum as well: https://github.com/ferd/ReVault/blob/5fede9f9459eff85db5643b5d5f37e2f3c7a1429/apps/revault/test/s3_integration_SUITE.erl#L195-L283 (do note also that under that mode with checksum, all part uploads need to be sequential)

onno-vos-dev closed this as completed Mar 24, 2022

vitorlimadev mentioned this issue Sep 6, 2023

chore: Add docs for multipart upload to S3 #167

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AWS.S3.complete_multipart_upload(client, bucket, key, %{}) returns an error #84

AWS.S3.complete_multipart_upload(client, bucket, key, %{}) returns an error #84

gabrielmancini commented Jul 26, 2021

philss commented Jul 27, 2021

gabrielmancini commented Aug 2, 2021 •

edited

Loading

dmorn commented Mar 24, 2022

dmorn commented Mar 24, 2022

dmorn commented Mar 24, 2022

onno-vos-dev commented Mar 24, 2022 •

edited

Loading

dmorn commented Mar 24, 2022

onno-vos-dev commented Mar 24, 2022

dmorn commented Mar 24, 2022

dmorn commented Mar 24, 2022

dmorn commented Mar 24, 2022

onno-vos-dev commented Mar 24, 2022

dmorn commented Mar 24, 2022

dmorn commented Mar 24, 2022

onno-vos-dev commented Mar 24, 2022

dmorn commented Mar 24, 2022

ferd commented Apr 6, 2024

AWS.S3.complete_multipart_upload(client, bucket, key, %{}) returns an error #84

AWS.S3.complete_multipart_upload(client, bucket, key, %{}) returns an error #84

Comments

gabrielmancini commented Jul 26, 2021

philss commented Jul 27, 2021

gabrielmancini commented Aug 2, 2021 • edited Loading

dmorn commented Mar 24, 2022

dmorn commented Mar 24, 2022

dmorn commented Mar 24, 2022

onno-vos-dev commented Mar 24, 2022 • edited Loading

dmorn commented Mar 24, 2022

onno-vos-dev commented Mar 24, 2022

dmorn commented Mar 24, 2022

dmorn commented Mar 24, 2022

dmorn commented Mar 24, 2022

onno-vos-dev commented Mar 24, 2022

dmorn commented Mar 24, 2022

dmorn commented Mar 24, 2022

onno-vos-dev commented Mar 24, 2022

dmorn commented Mar 24, 2022

ferd commented Apr 6, 2024

gabrielmancini commented Aug 2, 2021 •

edited

Loading

onno-vos-dev commented Mar 24, 2022 •

edited

Loading