S3 Multipart Content-Length mismatch error #145

marctollin · 2016-07-14T18:04:00Z

Up until today I've been able to do multipart uploads using 1.4.0 version of s3proxy, suddenly I am getting the following errors uploading s3 streams within my tests. I've tried upgrading to 1.5.0 prerelease and downgrading to 1.3.0 but I'm getting the same error. Any debugging tips?

java.io.IOException: Content-Length mismatch, actual: 244334 expected: 244068 (Service: Amazon S3; Status Code: 500; Error Code: 500 java.io.IOException: Content-Length mismatch, actual: 244334 expected: 244068; Request ID: null)
com.amazonaws.services.s3.model.AmazonS3Exception: java.io.IOException: Content-Length mismatch, actual: 244334 expected: 244068 (Service: Amazon S3; Status Code: 500; Error Code: 500 java.io.IOException: Content-Length mismatch, actual: 244334 expected: 244068; Request ID: null)
    at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1378)
    at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:924)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:702)
    at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:454)
    at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:416)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:365)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3995)
    at com.amazonaws.services.s3.AmazonS3Client.doUploadPart(AmazonS3Client.java:3029)
    at com.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:3014)

The text was updated successfully, but these errors were encountered:

gaul · 2016-07-14T19:11:12Z

Can you try some different versions of the AWS SDK and perhaps using V2 authentication or disabling chunked encoding? If you submit a full test can I can investigate further.

marctollin · 2016-07-14T20:36:29Z

Thanks! So looks like

val s3 = new AmazonS3Client()
s3.setS3ClientOptions(S3ClientOptions.builder().disableChunkedEncoding().build())

did the trick for the tests. Really bizarre why this started happening. Appreciate the suggestion.

gaul · 2016-07-14T20:47:01Z

Ideally chunked encoding should work -- we even have a test for this. Can you give some more detail about the original symptoms, preferably a test case?

gaul · 2016-07-14T21:06:58Z

I expect chunk encoded requests to fail previous to 1.5.0 prerelease -- previously S3Proxy did not decode these leading to overlong payloads. However it should work from 1.5.0 onward.

smigfu · 2017-03-08T15:10:25Z

@andrewgaul: I have the same problem when I upgrade the aws-sdk-java from 1.11.15 to 1.11.16. And I think I have tracked the cause down to the change in AWSS3V4Signer (see aws/aws-sdk-java@15ca1bb). Previously in 1.11.15 the requests were being uploaded with chunked encoding which changed. Naturally we want the chunked encoding available, so disabling it is not really an option.

If I understand it correct s3proxy doesn't support chunked encoding with v4 signing? I have tried with the latest version of s3proxy 1.5.2.

gaul · 2017-03-08T19:52:28Z

@smigfu V4 signature should work for both chunked and unchunked encoding -- see AwsSdkTest for testAwsV4Signature and testAwsV4SignatureNonChunked. Can you share more details on your issue?

smigfu · 2017-03-09T15:57:52Z

Sorry, I think I had a false assumption above. The chunked encoding doesn't work if s3proxy is set to no authentication. We don't set any credentials to the AWS client (in the real world our buckets are protected by IAM roles). We are using s3proxy to execute our tests locally and not against a live backend. S3 is fine with our configuration (no credentials and chunked-encoding but s3proxy throws the error [1]). Our s3proxy config file looks like this:

s3proxy.endpoint=http://0.0.0.0:3128
s3proxy.authorization=none
s3proxy.ignore-unknown-headers=true
s3proxy.ignore-unknown-parameters=true
jclouds.provider=filesystem
jclouds.identity=identity
jclouds.credential=credential
jclouds.filesystem.basedir=./s3proxy/data-root

I haven't been able to reproduce this with the AwsSdkTest yet, because I couldn't set the TestUtils#startS3Proxy to not use authentication :-)

[1]

E 03-09 16:47:47.834 S3Proxy-20 o.j.b.config.LocalBlobStore:93 |::] An error occurred storing the new blob with name [15e66991-ee67-3ff3-74f6-3166a19542b5/70a7e3c2-ba3d-0995-2f4b-bed06227930b] to container [permanent-bucket].
java.io.IOException: Content-Length mismatch, actual: 182 expected: 10
	at org.jclouds.filesystem.strategy.internal.FilesystemStorageStrategyImpl.putBlob(FilesystemStorageStrategyImpl.java:475)
	at org.jclouds.blobstore.config.LocalBlobStore.putBlob(LocalBlobStore.java:763)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
	at com.sun.proxy.$Proxy39.putBlob(Unknown Source)
	at org.gaul.s3proxy.S3ProxyHandler.handlePutBlob(S3ProxyHandler.java:1674)
	at org.gaul.s3proxy.S3ProxyHandler.doHandle(S3ProxyHandler.java:606)
	at org.gaul.s3proxy.S3ProxyHandlerJetty.handle(S3ProxyHandlerJetty.java:63)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.eclipse.jetty.server.Server.handle(Server.java:499)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258)
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	at java.lang.Thread.run(Thread.java:745)
W 03-09 16:47:47.834 S3Proxy-20 o.e.jetty.server.HttpChannel:396 |::] /permanent-bucket/15e66991-ee67-3ff3-74f6-3166a19542b5/70a7e3c2-ba3d-0995-2f4b-bed06227930b
java.lang.RuntimeException: java.io.IOException: Content-Length mismatch, actual: 182 expected: 10
	at com.google.common.base.Throwables.propagate(Throwables.java:160)
	at org.jclouds.blobstore.config.LocalBlobStore.putBlob(LocalBlobStore.java:775)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
	at com.sun.proxy.$Proxy39.putBlob(Unknown Source)
	at org.gaul.s3proxy.S3ProxyHandler.handlePutBlob(S3ProxyHandler.java:1674)
	at org.gaul.s3proxy.S3ProxyHandler.doHandle(S3ProxyHandler.java:606)
	at org.gaul.s3proxy.S3ProxyHandlerJetty.handle(S3ProxyHandlerJetty.java:63)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.eclipse.jetty.server.Server.handle(Server.java:499)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258)
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Content-Length mismatch, actual: 182 expected: 10
	at org.jclouds.filesystem.strategy.internal.FilesystemStorageStrategyImpl.putBlob(FilesystemStorageStrategyImpl.java:475)
	at org.jclouds.blobstore.config.LocalBlobStore.putBlob(LocalBlobStore.java:763)
	... 17 common frames omitted

j-kidd · 2017-03-09T17:23:43Z

I was also having this issue, and your suggestion does appear to have solved my problem. Running S3proxy with AWS_V4 'authorization' works fine with SDK version 1.11.86, 'none' authorization gives me the error listed above. Just wanted to say thanks, saved me from wasting even more time investigating this

gaul · 2017-03-09T19:06:05Z

@smigfu Can you provide a complete test case? I set s3proxy.authorization=none in src/test/resources/s3proxy.conf but mvn test -Dtest=AwsSdkTest succeeds for me, including the non-chunked test.

smigfu · 2017-03-10T10:19:33Z

I have added a PR #199 (mainly to add comments). What I have found are three preconditions which must be present:

Don't set any credentials on the AwsS3Client
Use HTTP (it will work with HTTPS)
Use "none" as authorization

@j-kidd : does all of the above matches your scenario as well?

smigfu · 2017-03-10T10:22:48Z

(just to add, since from a security point of view it sounds like a nightmare not using HTTPS and no Credentials: All our buckets do use IAM roles and are only accessed from our EC2 instances, thus we can omit the HTTPS overhead and rely on the IAM security model :-) )

j-kidd · 2017-03-10T13:40:50Z

@smigfu

So I don't think I'm seeing your first point, I'm hitting this error when using 'none' authorization with or without credentials set on the client.

I can only get it working with the newer AWS SDK when setting keys in S3proxy (and setting auth to an actual protocol) and having credentials set in the client.

Also check authentication type correctly in the constructor to handle configurations with anonymous access but set identity and credential. References #145.

gaul · 2017-03-11T03:47:45Z

@smigfu @j-kidd Please test with latest master which supports chunked encoding with anonymous requests and checks anonymous configuration correctly, which now does the right then setting "none" authorization with credentials set.

j-kidd · 2017-03-13T10:22:00Z

@andrewgaul

Looks good to me, testing against an S3proxy running locally I can make requests with or without credentials with 'none' authorisation and with AWS_V2_OR_V4 creds are checked as expected.

References #145.

marctollin closed this as completed Jul 14, 2016

gaul added a commit that referenced this issue Mar 11, 2017

Interpret chunked encoding with anonymous access

8224f6f

Also check authentication type correctly in the constructor to handle configurations with anonymous access but set identity and credential. References #145.

gaul mentioned this issue Mar 11, 2017

Anonymous access tests #200

Closed

gaul pushed a commit that referenced this issue Jan 9, 2018

Test for chunked encoding with anonymous access

e1f6cf0

References #145.

gaul pushed a commit that referenced this issue Jan 9, 2018

Test for chunked encoding with anonymous access

24a3d64

References #145.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

S3 Multipart Content-Length mismatch error #145

S3 Multipart Content-Length mismatch error #145

marctollin commented Jul 14, 2016 •

edited by gaul

gaul commented Jul 14, 2016

marctollin commented Jul 14, 2016

gaul commented Jul 14, 2016

gaul commented Jul 14, 2016

smigfu commented Mar 8, 2017 •

edited

gaul commented Mar 8, 2017

smigfu commented Mar 9, 2017

j-kidd commented Mar 9, 2017

gaul commented Mar 9, 2017

smigfu commented Mar 10, 2017

smigfu commented Mar 10, 2017

j-kidd commented Mar 10, 2017

gaul commented Mar 11, 2017

j-kidd commented Mar 13, 2017

S3 Multipart Content-Length mismatch error #145

S3 Multipart Content-Length mismatch error #145

Comments

marctollin commented Jul 14, 2016 • edited by gaul

gaul commented Jul 14, 2016

marctollin commented Jul 14, 2016

gaul commented Jul 14, 2016

gaul commented Jul 14, 2016

smigfu commented Mar 8, 2017 • edited

gaul commented Mar 8, 2017

smigfu commented Mar 9, 2017

j-kidd commented Mar 9, 2017

gaul commented Mar 9, 2017

smigfu commented Mar 10, 2017

smigfu commented Mar 10, 2017

j-kidd commented Mar 10, 2017

gaul commented Mar 11, 2017

j-kidd commented Mar 13, 2017

marctollin commented Jul 14, 2016 •

edited by gaul

smigfu commented Mar 8, 2017 •

edited