Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Content-MD5 does not match object md5 #1842

Closed
2mars opened this issue May 6, 2024 · 7 comments
Closed

Content-MD5 does not match object md5 #1842

2mars opened this issue May 6, 2024 · 7 comments
Assignees
@afranken
Labels
bug

Comments

@2mars
Copy link

2mars commented May 6, 2024 •
edited

With 3.7.3 version my code now fails when I try to put file using amazon s3 client (putObject).

I have an exception (that I haven't before) :

12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> PUT /test-baguette/sentinel_epsg32720_classified.tiff HTTP/1.1
12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> Host: localhost:42908
12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> amz-sdk-invocation-id: 7401ca2c-2c72-648b-709f-f63902fead17
12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> amz-sdk-request: attempt=1;max=4
12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> amz-sdk-retry: 0/0/500
12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=foo/20240506/eu-central-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=f226c1fd2282d702057f66e67d3d1bf55fdd4db0891faa7d14adc5037648fd97
12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> Content-MD5: h5qk8jrzBOtj94WoBhDShw==
12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> Content-Type: image/tiff
12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> User-Agent: aws-sdk-java/1.12.712 Mac_OS_X/14.4.1 OpenJDK_64-Bit_Server_VM/17.0.2+8-86 java/17.0.2 vendor/Oracle_Corporation cfg/retry-mode/legacy cfg/auth-source#unknown
12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD
12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> X-Amz-Date: 20240506T103749Z
12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> x-amz-decoded-content-length: 18636
12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> Content-Length: 18811
12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> Connection: Keep-Alive
12:37:49.882 [main] DEBUG org.apache.http.headers -- http-outgoing-0 >> Expect: 100-continue
12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "PUT /test-baguette/sentinel_epsg32720_classified.tiff HTTP/1.1[\r][\n]"
12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "Host: localhost:42908[\r][\n]"
12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "amz-sdk-invocation-id: 7401ca2c-2c72-648b-709f-f63902fead17[\r][\n]"
12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "amz-sdk-request: attempt=1;max=4[\r][\n]"
12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=foo/20240506/eu-central-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=f226c1fd2282d702057f66e67d3d1bf55fdd4db0891faa7d14adc5037648fd97[\r][\n]"
12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "Content-MD5: h5qk8jrzBOtj94WoBhDShw==[\r][\n]"
12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "Content-Type: image/tiff[\r][\n]"
12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "User-Agent: aws-sdk-java/1.12.712 Mac_OS_X/14.4.1 OpenJDK_64-Bit_Server_VM/17.0.2+8-86 java/17.0.2 vendor/Oracle_Corporation cfg/retry-mode/legacy cfg/auth-source#unknown[\r][\n]"
12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD[\r][\n]"
12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "X-Amz-Date: 20240506T103749Z[\r][\n]"
12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "x-amz-decoded-content-length: 18636[\r][\n]"
12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "Content-Length: 18811[\r][\n]"
12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "Expect: 100-continue[\r][\n]"
12:37:49.882 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "[\r][\n]"
12:37:49.900 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "HTTP/1.1 100 Continue[\r][\n]"
12:37:49.900 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "[\r][\n]"
12:37:49.901 [main] DEBUG org.apache.http.headers -- http-outgoing-0 << HTTP/1.1 100 Continue
12:37:49.902 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "48cc;chunk-signature=ee792a1a7e167e4fa48767766c31422d2f7009e88217189abe8e6b75e9996fb2[\r][\n]"
12:37:49.902 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "II*[0x0][0x8][0x0][0x0][0x0][0x11][0x0][0x0][0x1][0x3][0x0][0x1][0x0][0x0][0x0]B[0x0][0x0][0x0][0x1][0x1][0x3][0x0][0x1][0x0][0x0][0x0]D[0x0][0x0][0x0][0x2][0x1][0x3][0x0][0x1][0x0][0x0][0x0] [0x0][0x0][0x0][0x3][0x1][0x3][0x0][0x1][0x0][0x0][0x0][0x1][0x0][0x0][0x0][0x6][0x1][0x3][0x0][0x1][0x0][0x0][0x0][0x1][0x0][0x0][0x0][0x11][0x1][0x4][0x0][0x3][0x0][0x0][0x0][0xe6][0x0][0x0][0x0][0x15][0x1][0x3][0x0][0x1][0x0][0x0][0x0][0x1][0x0][0x0][0x0][0x16][0x1][0x3][0x0][0x1][0x0][0x0][0x0][0x1f][0x0][0x0][0x0][0x17][0x1][0x4][0x0][0x3][0x0][0x0][0x0][0xda][0x0][0x0][0x0][0x1c][0x1][0x3][0x0][0x1][0x0][0x0][0x0][0x1][0x0][0x0][0x0]S[0x1][0x3][0x0][0x1][0x0][0x0][0x0][0x2][0x0][0x0][0x0][0xe][0x83][0xc][0x0][0x3][0x0][0x0][0x0][0x6][0x2][0x0][0x0][0x82][0x84][0xc][0x0][0x6][0x0][0x0][0x0][0x1e][0x2][0x0][0x0][0xaf][0x87][0x3][0x0] [0x0][0x0][0x0]N[0x2][0x0][0x0][0xb1][0x87][0x2][0x0][0x1e][0x0][0x0][0x0][0x8e][0x2][0x0][0x0][0x80][0xa4][0x2][0x0][0x14][0x1][0x0][0x0][0xf2][0x0][0x0][0x0][0x81][0xa4][0x2][0x0][0x3][0x0][0x0][0x0]-1[0x0][0x0][0x0][0x0][0x0][0x0][0xf8][0x1f][0x0][0x0][0xf8][0x1f][0x0][0x0]0[0x6][0x0][0x0][0xac][0x2][0x0][0x0][0xa4]"[0x0][0x0][0x9c]B[0x0][0x0][\n]"
12:37:49.902 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> " 2[\n]"
12:37:49.902 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> " -0.41644385026738[\n]"
12:37:49.902 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> " -1[\n]"
12:37:49.902 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> " 0.93894098334153[\n]"
12:37:49.902 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "[\n]"
12:37:49.904 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "[0x0][0x0][0x0][0x0][0x0][0x0][0x0]$@[0x0][0x0] ..."
12:37:49.906 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "[0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0] ... [\r][\n]"
12:37:49.906 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "0;chunk-signature=804a6f3d7ff45f1bc11eec20f425c5fdb9d86609a19744aacf1c0d6a300a1f4b[\r][\n]"
12:37:49.906 [main] DEBUG org.apache.http.wire -- http-outgoing-0 >> "[\r][\n]"
12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "HTTP/1.1 400 Bad Request[\r][\n]"
12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "Server: Jetty(12.0.8)[\r][\n]"
12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "Date: Mon, 06 May 2024 10:37:49 GMT[\r][\n]"
12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "Vary: Origin[\r][\n]"
12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "Vary: Access-Control-Request-Method[\r][\n]"
12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "Vary: Access-Control-Request-Headers[\r][\n]"
12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "Transfer-Encoding: chunked[\r][\n]"
12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "[\r][\n]"
12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "84[\r][\n]"
12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "BadRequestContent-MD5 does not match object md5"
12:37:49.969 [main] DEBUG org.apache.http.headers -- http-outgoing-0 << HTTP/1.1 400 Bad Request
12:37:49.969 [main] DEBUG org.apache.http.headers -- http-outgoing-0 << Server: Jetty(12.0.8)
12:37:49.969 [main] DEBUG org.apache.http.headers -- http-outgoing-0 << Date: Mon, 06 May 2024 10:37:49 GMT
12:37:49.969 [main] DEBUG org.apache.http.headers -- http-outgoing-0 << Vary: Origin
12:37:49.969 [main] DEBUG org.apache.http.headers -- http-outgoing-0 << Vary: Access-Control-Request-Method
12:37:49.969 [main] DEBUG org.apache.http.headers -- http-outgoing-0 << Vary: Access-Control-Request-Headers
12:37:49.969 [main] DEBUG org.apache.http.headers -- http-outgoing-0 << Content-Type: application/xml
12:37:49.969 [main] DEBUG org.apache.http.headers -- http-outgoing-0 << Transfer-Encoding: chunked
12:37:49.969 [main] DEBUG org.apache.http.impl.execchain.MainClientExec -- Connection can be kept alive for 60000 MILLISECONDS
12:37:49.969 [main] DEBUG com.amazonaws.retry.ClockSkewAdjuster -- Reported server date (from 'Date' header): Mon, 06 May 2024 10:37:49 GMT
12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "[\r][\n]"
12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "0[\r][\n]"
12:37:49.969 [main] DEBUG org.apache.http.wire -- http-outgoing-0 << "[\r][\n]"
12:37:49.969 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager -- Connection [id: 0][route: {}->http://localhost:42908] can be kept alive for 60.0 seconds
12:37:49.969 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection -- http-outgoing-0: set socket timeout to 0
12:37:49.969 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager -- Connection released: [id: 0][route: {}->http://localhost:42908][total available: 1; route allocated: 1 of 50; total allocated: 1 of 50]
12:37:49.970 [main] DEBUG com.amazonaws.request -- Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Content-MD5 does not match object md5 (Service: Amazon S3; Status Code: 400; Error Code: BadRequest; Request ID: null; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null
12:37:49.970 [main] DEBUG com.amazonaws.auth.AwsChunkedEncodingInputStream -- AwsChunkedEncodingInputStream reset (will reset the wrapped stream because it is mark-supported).

com.amazonaws.services.s3.model.AmazonS3Exception: Content-MD5 does not match object md5 (Service: Amazon S3; Status Code: 400; Error Code: BadRequest; Request ID: null; S3 Extended Request ID: null; Proxy: null)
, S3 Extended Request ID: null
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1880)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5558)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5505)
at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:423)
at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6639)
at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1892)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1852)
@afranken afranken self-assigned this May 6, 2024
@afranken
Copy link
Member

afranken commented May 6, 2024

@2mars
this is probably related to #1840

Do you have a working test for me to reproduce this?

@2mars
Copy link
Author

2mars commented May 6, 2024 

@Test
void fail_md5() throws URISyntaxException {
    AmazonS3 s3client = AmazonS3ClientBuilder.standard()
            .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials("foo", "bar")))
            .withEndpointConfiguration(new EndpointConfiguration("http://localhost:42908", "eu-central-1"))
            .enablePathStyleAccess().build();
    Bucket someTestBucket = s3client.createBucket("test-baguette");
    File someGeotiff = new File(this.getClass().getResource("/sentinel_epsg32720_classified.tiff").toURI());
    s3client.putObject(new PutObjectRequest(someTestBucket.getName(), someGeotiff.getName(), someGeotiff));
    assertThat(s3client.getObject(new GetObjectRequest(someTestBucket.getName(), someGeotiff.getName()))).isNotNull();
}

@afranken
Copy link
Member

afranken commented May 6, 2024

thanks!

Looks like this only happens when using http as a connection method (which results in signed chunked uploads) in connection to uploading binary files like images.
In our Integration-Tests and unit-tests we're using text files for testing.
The Integration-Tests use https as a connection method with few exceptions.

That's why this issue was not caught during the refactorings I made for 3.7.3.
I'll look into a solution.

@afranken afranken mentioned this issue May 6, 2024
Closed
@jonasholtkamp
Copy link

jonasholtkamp commented May 8, 2024 •
edited

I'm having the same error with HTTP, however I'm uploading a plaintext JSON file. File size is ~3.5k, using a pretty vanilla container instance.

@afranken afranken added the bug label May 12, 2024
afranken added a commit that referenced this issue May 12, 2024
@afranken
Skip multiple CRLF if present.
adb47e0 
For unknown reasons, when using the async clients to upload chunked
data, the SDK sometimes inserts multiple CRLFs between chunks.
Skip if we need to.

Fixes #1840
Fixes #1842
afranken added a commit that referenced this issue May 12, 2024
@afranken
More tests with different files, sync/async and http/https
ae809de 
For unknown reasons, the AWS Java SDKv2 sends the wrong checksum for
SHA256 when using HTTP. Skip this test for now.

Fixes #1840
Fixes #1842
afranken added a commit that referenced this issue May 12, 2024
@afranken
More tests with different files, sync/async and http/https
4b679ee 
For unknown reasons, the AWS Java SDKv2 sends the wrong checksum for
SHA256 when using HTTP. Skip this test for now.

Fixes #1840
Fixes #1842
afranken added a commit that referenced this issue May 12, 2024
@afranken
More tests with different files, sync/async and http/https
6cb76c7 
For unknown reasons, the AWS Java SDKv2 sends the wrong checksum for
SHA256 when using HTTP. Skip this test for now.

Fixes #1840
Fixes #1842
@afranken afranken mentioned this issue May 12, 2024
Merged
2 tasks
@afranken
Copy link
Member

@jonasholtkamp @2mars
I released 3.8.0 earlier today which should fix the problems in most cases.
Only the combination http / sync / sha256 does not work, at least with the AWS v2 SDK for Java.

@jonasholtkamp
Copy link

Upgraded to 3.8.0, everything works like it did in 3.7.2, thank you.

One extra question, might be a little offtopic: How does one set up the HTTPS client with S3Mock? The README mentions port 9191. If I shoot HTTPS request against that port, I get these errors:

software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

I suppose I need to trust a certificate first? Do I need to generate, self-sign and install one with the mock? Some guidance would be appreciated.

@afranken
Copy link
Member

@jonasholtkamp
S3Mock bring its own, non-compliant SSL certificate.
The HttpClient used by the AWS SDK can be configured to ignore that. The integration-test module contains setups for a number of HttpClients, for example:
https://github.com/adobe/S3Mock/blob/main/integration-tests/src/test/kotlin/com/adobe/testing/s3mock/its/S3TestBase.kt#L156

If you run S3Mock in Docker, you could bring your own, valid SSL certificate, see #281

I'll see if I can add something to the README.md about using SSL in a later release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@afranken afranken

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@afranken @2mars @jonasholtkamp