Can't putObject with path as a key

[bamthomas]

Hi all,

I can't do a putObject with a path key :

  @Test
  public void shouldUploadObjectWithAPath() throws Exception {
        final File uploadFile = new File(UPLOAD_FILE_NAME);

        s3Client.createBucket(BUCKET_NAME);
        s3Client.putObject(new PutObjectRequest(BUCKET_NAME,UPLOAD_FILE_NAME, uploadFile));

        assertThat(s3Client.doesObjectExist(BUCKET_NAME, UPLOAD_FILE_NAME), is(true));
  }

I have a Not Acceptable 406 code.
When I test this with the real S3, it creates a file in a src/test/resources/ path.

am I missing something ?

[timoe]

Thanks for that @bamthomas.
This is caused by the path mapping in FileStoreController#putObject where mapping is not able to handle / separators within the key.

[magro]

@timoe Are you referring to the request mapping "/{bucketName}/{fileName:.+}"? We're running into this as well, and the question is how easily/fast this can be fixed (as everybody else we're on an extremely tight schedule right now ;-)).

[timoe]

@magro: Right, that’s what I assume.

We have to

• play around with that mapping pattern and find a better one or
• include the buckets name only and get remaining path from the Request URI

However, we´re going to work on that.

[magro]

@timoe Ok. Because this is that urgent for us we'll now try to fix this. In the good case that we can come up with a PR, how fast could you release it? Mit Grüßen aus der Schanze :-)

[timoe]

@magro: I played around with mapping patterns but got stuck for some methods. So I reworked mappings for extracting the object key from request URI.

I already opened a PR for it based on this branch https://github.com/timoe/S3Mock/tree/fix/put_objects_including_path_fragments. You might to take a look at it and give it a try on your side.

Grüße vom Fischmarkt.

[mattelacchiato]

I'll have a look

[mattelacchiato]

There's currently no error returning for requests with slashes in the key. But a simple PUT for an object and the list this object does not work:

14:41:09.401 DEBUG com.amazonaws.metrics.AwsSdkMetrics - Admin mbean registered under com.amazonaws.management:type=AwsSdkMetrics
/55/56/57/5859.eZx
14:41:09.711 DEBUG com.amazonaws.request - Sending Request: PUT http://localhost:9090 /jMqeFxugxr/55/56/57/5859.eZx Headers: (User-Agent: aws-sdk-java/1.11.241 Linux/4.8.0-58-generic OpenJDK_64-Bit_Server_VM/25.151-b12 java/1.8.0_151 scala/2.11.11 groovy/2.4.3, amz-sdk-invocation-id: b3ea2bd4-ebfc-8e9e-3224-c3a07fe09924, Content-Length: 21, Content-Type: application/octet-stream, ) 
14:41:09.743 DEBUG com.amazonaws.auth.AWS4Signer - AWS4 Canonical Request: '"PUT
/jMqeFxugxr/55/56/57/5859.eZx

amz-sdk-invocation-id:b3ea2bd4-ebfc-8e9e-3224-c3a07fe09924
amz-sdk-retry:0/0/500
content-length:194
content-type:application/octet-stream
host:localhost:9090
user-agent:aws-sdk-java/1.11.241 Linux/4.8.0-58-generic OpenJDK_64-Bit_Server_VM/25.151-b12 java/1.8.0_151 scala/2.11.11 groovy/2.4.3
x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD
x-amz-date:20171206T134109Z
x-amz-decoded-content-length:21

amz-sdk-invocation-id;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length
STREAMING-AWS4-HMAC-SHA256-PAYLOAD"
14:41:09.743 DEBUG com.amazonaws.auth.AWS4Signer - AWS4 String to Sign: '"AWS4-HMAC-SHA256
20171206T134109Z
20171206/us-east-1/s3/aws4_request
ec4e29c16a19cd52d0662112d49323f5206b3abb31c872d3a0e3502582f38eae"
14:41:09.744 DEBUG com.amazonaws.auth.AWS4Signer - Generating a new signing key as the signing key not available in the cache for the date 1512518400000
14:41:09.872 TRACE c.a.a.AwsChunkedEncodingInputStream - 108 byte read from the stream.
14:41:09.872 TRACE c.a.a.AwsChunkedEncodingInputStream - 86 byte read from the stream.
14:41:09.956 DEBUG com.amazonaws.request - Received successful response: 201, AWS Request ID: null
14:41:09.956 DEBUG com.amazonaws.requestId - x-amzn-RequestId: not available
14:41:09.956 DEBUG com.amazonaws.requestId - AWS Request ID: not available
list objects for jMqeFxugxr and 55/56/57/5859.eZx
14:41:10.058 DEBUG com.amazonaws.request - Sending Request: GET http://localhost:9090 /jMqeFxugxr/ Parameters: ({"list-type":["2"],"prefix":["55/56/57/5859.eZx"],"fetch-owner":["false"]}Headers: (User-Agent: aws-sdk-java/1.11.241 Linux/4.8.0-58-generic OpenJDK_64-Bit_Server_VM/25.151-b12 java/1.8.0_151 scala/2.11.11 groovy/2.4.3, amz-sdk-invocation-id: 0b4dfa56-bf35-f805-854b-6140241130e9, Content-Type: application/octet-stream, ) 
14:41:10.059 DEBUG com.amazonaws.auth.AWS4Signer - AWS4 Canonical Request: '"GET
/jMqeFxugxr/
fetch-owner=false&list-type=2&prefix=55%2F56%2F57%2F5859.eZx
amz-sdk-invocation-id:0b4dfa56-bf35-f805-854b-6140241130e9
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:9090
user-agent:aws-sdk-java/1.11.241 Linux/4.8.0-58-generic OpenJDK_64-Bit_Server_VM/25.151-b12 java/1.8.0_151 scala/2.11.11 groovy/2.4.3
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20171206T134110Z

amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
14:41:10.059 DEBUG com.amazonaws.auth.AWS4Signer - AWS4 String to Sign: '"AWS4-HMAC-SHA256
20171206T134110Z
20171206/us-east-1/s3/aws4_request
b9e6b6716cfa8937d8bbc8504bede65677e7b60bea4711faf7dbde3b913cfabc"

[timoe]

@mattelacchiato: Can you please add a code snippet / unit test for that.

I tried listing bucket objects inside a folder and failed. As I did updates on it: Can you please pull the branch I mentioned again. It contains an update regarding visiting the directory hierarchy within the bucket which hopefully solves that problem.

[mattelacchiato]

Yeah, it works! When will you release it? Do you still need a test for that?

[agudian]

Great, will roll the release now...

[magro]

Awesome, thanks @timoe and @agudian !

[agudian]

We had some issues with the release, so there was a bit of a delay. Version 1.0.2 is now pushed to Docker-Hub and to Maven-Central.