Gzip compression doesn't work with DynamoDB (crc32 errors) #526

jakozaur · 2015-10-05T10:33:32Z

If you setup gzip, sometimes I get an exception:

Caused by: com.amazonaws.internal.CRC32MismatchException: Client calculated crc32 checksum didn't match that calculated by server side
    at com.amazonaws.http.JsonResponseHandler.handle(JsonResponseHandler.java:112)
    at com.amazonaws.http.JsonResponseHandler.handle(JsonResponseHandler.java:42)
    at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1072)
    at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:746)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)

Android SDK fixed that bug. See:
aws-amplify/aws-sdk-android#40
and the actual fix:
aws-amplify/aws-sdk-android@b03c51e

Please also fix it in Java SDK.

The text was updated successfully, but these errors were encountered:

sarxos · 2015-11-10T11:05:37Z

Hi @aws team,

We are also impacted by this bug. Please fix this 👍

ohookins · 2015-12-22T10:50:52Z

I've also noticed this while trying to use it with Kinesis streaming. Since the easiest way to set up the KCL is with a common client for Kinesis, DynamoDB and CloudWatch, it means that you need to disable Gzip compression for all three of those, which is not ideal.

2015-12-22_10:43:25.44544 Dec 22, 2015 10:43:25 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncTask call
2015-12-22_10:43:25.44546 SEVERE: Caught exception while sync'ing Kinesis shards and leases
2015-12-22_10:43:25.44547 com.amazonaws.services.kinesis.leases.exceptions.DependencyException: com.amazonaws.AmazonClientException: Unable to execute HTTP request: Client calculated crc32 checksum didn't match that calculated by server side
2015-12-22_10:43:25.44548   at com.amazonaws.services.kinesis.leases.impl.LeaseManager.list(LeaseManager.java:268)
2015-12-22_10:43:25.44549   at com.amazonaws.services.kinesis.leases.impl.LeaseManager.listLeases(LeaseManager.java:203)
2015-12-22_10:43:25.44549   at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncer.syncShardLeases(ShardSyncer.java:119)
2015-12-22_10:43:25.44550   at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncer.checkAndCreateLeasesForNewShards(ShardSyncer.java:88)
2015-12-22_10:43:25.44551   at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncTask.call(ShardSyncTask.java:68)
2015-12-22_10:43:25.44551   at com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:49)
2015-12-22_10:43:25.44553   at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.initialize(Worker.java:383)
2015-12-22_10:43:25.44553   at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.run(Worker.java:318)
2015-12-22_10:43:25.44554   at com.X.Y.Z.main(Z.java:137)
2015-12-22_10:43:25.44555 Caused by: com.amazonaws.AmazonClientException: Unable to execute HTTP request: Client calculated crc32 checksum didn't match that calculated by server side
2015-12-22_10:43:25.44557   at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:587)
2015-12-22_10:43:25.44568   at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:362)
2015-12-22_10:43:25.44569   at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:328)
2015-12-22_10:43:25.44570   at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:307)
2015-12-22_10:43:25.44570   at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1805)
2015-12-22_10:43:25.44571   at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.scan(AmazonDynamoDBClient.java:1525)
2015-12-22_10:43:25.44572   at com.amazonaws.services.kinesis.leases.impl.LeaseManager.list(LeaseManager.java:227)
2015-12-22_10:43:25.44573   ... 8 more
2015-12-22_10:43:25.44575 Caused by: com.amazonaws.internal.CRC32MismatchException: Client calculated crc32 checksum didn't match that calculated by server side
2015-12-22_10:43:25.44576   at com.amazonaws.http.JsonResponseHandler.handle(JsonResponseHandler.java:113)
2015-12-22_10:43:25.44576   at com.amazonaws.http.JsonResponseHandler.handle(JsonResponseHandler.java:42)
2015-12-22_10:43:25.44577   at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1142)
2015-12-22_10:43:25.44589   at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:853)
2015-12-22_10:43:25.44592   at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:576)
2015-12-22_10:43:25.44594   ... 14 more

zhangzhx · 2015-12-22T18:30:12Z

It looks like a bug in java sdk, will fix that. Thanks for the report.

zhangzhx · 2015-12-22T18:46:19Z

Hello @jakozaur , can you give us a specific test case ? I am working on fixing this.

jakozaur · 2015-12-23T09:36:13Z

@zhangzhx I'm hitting it every time, when I enable Gzip compression and than try to use DynamoDB Api.

I just created DynamoDB client:

new AmazonDynamoDBClient(credentials, new ClientConfiguration().withGzip(true))

and then every network call fails.

fernomac · 2015-12-23T13:49:44Z

aws/aws-sdk-js#405 is probably relevant to your interests. DynamoDB is returning a checksum of the compressed bytes, which many HTTP clients transparently decompress for you making the checksum "wrong". Ideally they would switch to returning a checksum over the uncompressed bytes, but until that happens the only real option is to disable checksum validation when you might get a gzipped response. :(

fernomac · 2015-12-23T14:33:28Z

Oh bother, didn't see the linked Android SDK issue. It's unfortunate that they fixed this client-side and are now depending on the checksum matching the compressed bytes, as that pretty much closes the door on a fix coming from the server-side. If you have a reasonable way of getting the compressed bytes out of the Apache HttpClient, you may as well follow suit and verify the checksum on that.

zhangzhx · 2015-12-23T20:34:46Z

Thanks @fernomac for the explain.

I will reach out to the DynamoDB server team and figure out a safe way to fix this and not breaking android as well. Maybe letting them send both CRCs from uncompressed and compressed payload would be an option.

iCesofT · 2016-03-31T11:09:28Z

@zhangzhx, could you please let us know if there is any update? Thank you in advance.

bcarl · 2016-04-16T01:01:18Z

Also hitting this. Any status on a fix?

zhangzhx · 2016-04-21T18:16:16Z

Sorry for the long wait. We asked the service team and they are not planning to change the checksum to uncompressed data. We will soon migrate our underlying Apache Httpclient from 4.3.6 to a newer version 4.5. With that said, we will fix this issue in the new Httpclient by disabling response content automatic decompression for dynamo. I will let you know once that happens.

jorgebsa · 2016-05-17T18:52:59Z

+1 :)

zhangzhx · 2016-07-22T00:33:27Z

We have fixed this bug in our latest release 1.11.20. Please try the new version and let us know if you have further questions. Sorry for the long waiting of the bug fix.

theRealDrumBum · 2016-07-22T23:44:05Z

I think you actually may have introduced a bigger bug.... out of the blue once we updated to 1.11.20 we started seeing this issue (when previously we were not).

zhangzhx · 2016-07-22T23:49:33Z

Hi @theRealDrumBum, can you give me a detailed description what the issue you are seeing?

theRealDrumBum · 2016-07-22T23:57:16Z

Its basically the same issue reported above, except introduced with 1.11.20...

stack-trace.txt

zhangzhx · 2016-07-23T00:05:32Z

Can you give me some sample code to reproduce this bug? Thanks.

breedloj · 2016-07-25T19:40:47Z

Any details about how you've configured the client would be very helpful as well.

thisismana · 2016-07-29T18:45:38Z

I can help out, we are having the same issue. Version 1.11.21 in a spring-boot 1.4.rc1 project.

In a Spring app:

@Configuration
public class AwsConfig {

    @Bean
    public AWSCredentialsProvider awsCredentialsProvider() {
        return new DefaultAWSCredentialsProviderChain();
    }

    @Bean
    public AmazonDynamoDBAsyncClient client(final AWSCredentialsProvider credentialsProvider,
                                            final DynamoDBConfig dynamoDBConfig) {
        final ClientConfiguration configuration = new ClientConfiguration();
        configuration.setRequestTimeout(dynamoDBConfig.getTimeoutInMillis());
        configuration.setClientExecutionTimeout(dynamoDBConfig.getTimeoutInMillis());
        final AmazonDynamoDBAsyncClient client = new AmazonDynamoDBAsyncClient(credentialsProvider, configuration);
        client.setRegion(Region.getRegion(Regions.fromName(dynamoDBConfig.getRegion())));
        return client;
    }
}

And somewhere else, the exception mentioned above when executing:

GetItemResult getItemRes = client.getItem(
                new GetItemRequest(settingsTable, ImmutableMap.of(SETTINGS_ID_KEY, new AttributeValue().withS(SETTINGS_NEWER_URL_ID)))
        );

Exception:

com.amazonaws.AmazonClientException: Unable to execute HTTP request: Client calculated crc32 checksum didn't match that calculated by server side
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:735)
    at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:475)
    at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:437)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:386)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:2074)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:2044)
    at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.getItem(AmazonDynamoDBClient.java:1392)
Caused by: com.amazonaws.internal.CRC32MismatchException: Client calculated crc32 checksum didn't match that calculated by server side
    at com.amazonaws.http.JsonResponseHandler.handle(JsonResponseHandler.java:125)
    at com.amazonaws.http.JsonResponseHandler.handle(JsonResponseHandler.java:46)
    at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1274)
    at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:913)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:723)

shorea · 2016-07-29T18:56:28Z

Which versions of the aws-java-sdk-dynamodb and aws-java-sdk-core are being used at runtime?

thisismana · 2016-07-30T21:04:06Z

We got com.amazonaws:aws-java-sdk-dynamodb and com.amazonaws:aws-java-sdk-sqs on the dependency list. Those are on the classpath at runtime:

com.amazonaws:aws-java-sdk-core:1.11.21
com.amazonaws:aws-java-sdk-dynamodb:1.11.21
com.amazonaws:aws-java-sdk-kms:1.11.21
com.amazonaws:aws-java-sdk-s3:1.11.21
com.amazonaws:aws-java-sdk-sqs:1.11.21

thisismana · 2016-08-01T07:04:28Z

Still the same with 1.11.22, but downgrading to 1.11.19 avoids this problem. Anything new on this topic?

shorea · 2016-08-01T16:09:18Z

I have not been able to reproduce this. Would it be possible to provide some wire log samples? Keep in mind logs may contain sensitive data.

http://docs.aws.amazon.com/java-sdk/latest/developer-guide/java-dg-logging.html

zhangzhx · 2016-08-01T19:10:29Z

Hey @thisismana, do you see this error not having Gzip configured? I couldn't reproduce this issue as well. Please provide some wire log for us to further investigate. Thanks.

thisismana · 2016-08-05T10:16:56Z

Sorry for the late answer, but the problem seems to be resolved now. Even with the versions mentioned above. Guess it wasn't the java-sdks fault.

retested it with 1.11.21 and 1.11.22, as well as the new 1.11.23

close the issue?

shorea · 2016-08-05T16:02:35Z

Hmm interesting. Well if you see any further issues with it please let us know.

danvallejo · 2016-08-16T20:35:51Z

It's still an issue.

zhangzhx · 2016-08-17T00:05:42Z

Can you give us more information such as sample code that can reproduce this issue, error log or exception trace? Thanks.

danvallejo · 2016-08-17T18:07:03Z

com.amazonaws.AmazonClientException: Unable to execute HTTP request: Client calculated crc32 checksum didn't match that calculated by server side
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:732)
at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:475)
at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:437)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:386)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:2078)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:2048)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.query(AmazonDynamoDBClient.java:1690)
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper.query(DynamoDBMapper.java:1649)
at com.nordstrom.amp.core.dao.mappers.HashKeyMapper.query(HashKeyMapper.java:145)
at com.nordstrom.amp.core.dao.mappers.WebReadyAssetsMapper.queryByAssetId(WebReadyAssetsMapper.java:193)
at com.nordstrom.amp.core.dao.mappers.WebReadyAssetsMapper.getItemByAssetId(WebReadyAssetsMapper.java:118)
at com.nordstrom.amp.core.dao.mappers.WebReadyAssetsMapper.getItemByAssetId(WebReadyAssetsMapper.java:105)
at com.nordstrom.amp.core.repositories.WebReadyAssetRepository.getWebReadyAsset(WebReadyAssetRepository.java:53)
at com.nordstrom.amp.migration.Map.map(Map.java:47)
at com.nordstrom.amp.migration.Map.map(Map.java:21)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
Caused by: com.amazonaws.internal.CRC32MismatchException: Client calculated crc32 checksum didn't match that calculated by server side
at com.amazonaws.http.JsonResponseHandler.handle(JsonResponseHandler.java:125)
at com.amazonaws.http.JsonResponseHandler.handle(JsonResponseHandler.java:46)
at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1277)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:916)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:723)
... 18 more

shorea · 2016-08-17T18:22:52Z

Do you see this for every request?

danvallejo · 2016-08-17T18:50:30Z

Here's the code. I've stripped out as much as I can.
Yes, every single time.

java -jar build\libs\word-count-emr-1.0-adee88b.jar
WordCountEMRTest.zip

danvallejo · 2016-08-17T20:38:33Z

I've stripped out even more of our code.
WordCountEMRTest-2.zip

danvallejo · 2016-08-17T20:42:45Z

1.11.19 works
1.11.20 fails with the crc32 error

zhangzhx · 2016-08-17T20:47:59Z

Thanks for providing these sample code. I saw you intended to perform a query operation using DynamoDB Mapper. I couldn't reproduce this issue. Could you turn on log4j to get the wire log for us to investigate? Also, did you turn on Gzip in the ClientConfiguration setting?

Here is how to turn on log4j. Keep in mind logs may contain sensitive data such as credentials.
http://docs.aws.amazon.com/java-sdk/latest/developer-guide/java-dg-logging.html

danvallejo · 2016-08-17T20:58:16Z

Gzip doesn't affect the crc error.

I'll try the wire log...

public ClientConfiguration getClientConfiguration() {
    if (clientConfiguration == null) {
        clientConfiguration = new ClientConfiguration();
        clientConfiguration.setUseGzip(true);
        clientConfiguration.setConnectionTimeout(clientConnectionTimeout);
        clientConfiguration.setClientExecutionTimeout(clientExecutionTimeout);
        clientConfiguration.setUseGzip(true); // true or false doesn't affect the crc error
        logger.info("getClientConfiguration");

        if (!StringUtils.isEmpty(proxyHost) && !StringUtils.isEmpty(proxyPort)) {
            clientConfiguration.setProxyHost(proxyHost);
            clientConfiguration.setProxyPort(proxyPort);
        }
    }
    return clientConfiguration;
}

danvallejo · 2016-08-17T22:32:56Z

@zhangzhx can you email me at dan.vallejo / nordstrom.com

shorea · 2016-08-17T22:42:27Z

Able to reproduce with the sample you've provided. Digging into it now.

danvallejo · 2016-08-17T22:43:11Z

WOOHOO

shorea · 2016-08-17T22:55:38Z

For some reason it's not wrapping the content in a CRC calculating input stream so the client side crc is always 0. Still investigating.

shorea · 2016-08-17T23:05:24Z

Ah okay got it. It's an issue with the request timeout feature not playing nice with how we are calculating the checksum now. For the request timeout when wrap with a BufferedHttpEntity to ensure we read the entire request before leaving AmazonHttpClient (to strictly enforce the timeout over the whole receiving of the request) and this is messing up checksum calculation. Looking for a fix now.

shorea · 2016-08-17T23:25:15Z

Okay got a fix in mind and I'll make sure it goes out in tomorrows release. If this is blocking you then temporarily disabling the request timeout and client execution timeout features will fix this issue.

shorea · 2016-08-18T00:20:36Z

@danvallejo Thanks for reporting this and providing the sample code, it was enormously helpful tracking this issue down. The fix has been staged and will be released tomorrow as part of 1.11.28.

theRealDrumBum · 2016-08-18T00:44:48Z

Sweet! I think Dan deserves an Xl Dynamodb shirt - we can meet you in the
lobby of Blackfoot!

-Matt

On Wed, Aug 17, 2016 at 5:20 PM Andrew Shore notifications@github.com
wrote:

@danvallejo https://github.com/danvallejo Thanks for reporting this and
providing the sample code, it was enormously helpful tracking this issue
down. The fix has been staged and will be released tomorrow as part of
1.11.28.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#526 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPN80MOxCuzzqaRZR0i4e3mv25eJwdUks5qg6VfgaJpZM4GI4i8
.

danvallejo · 2016-08-18T01:08:28Z

WOOOOOOOOHOOOOOOOOOOOOO

danvallejo · 2016-08-19T15:47:38Z

I still don't see the 1.11.28 package.
https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-core

shorea · 2016-08-19T15:52:23Z

Not sure how frequently that's synced with Maven Central. You should be able to pull it down from central though.

http://search.maven.org/#artifactdetails|com.amazonaws|aws-java-sdk-core|1.11.28|jar

danvallejo · 2016-08-19T16:01:38Z

What public maven url do you use?

buildscript {
ext {
AWSSDK = '1.11.28'
}
repositories {
maven { url "https://mvnrepo.nordstrom.net/nexus/content/groups/public/" }
}
}

shorea · 2016-08-19T16:07:51Z

Maven Central is http://repo1.maven.org/maven2/ I believe

danvallejo · 2016-08-19T17:56:06Z

Fix VERIFIED. Thanks.

zhangzhx added the Will fix label Dec 22, 2015

zhangzhx self-assigned this Dec 22, 2015

zhangzhx added the waiting-reply label Dec 22, 2015

zhangzhx added waiting-service-reply and removed waiting-reply labels Dec 23, 2015

zrzka mentioned this issue Jul 11, 2016

crc32-mismatch for big requests nervous-systems/hildebrand#12

Closed

zhangzhx closed this as completed Jul 22, 2016

debora-ito added service-api This issue is due to a problem in a service API, not the SDK implementation. and removed waiting-service-reply labels Feb 24, 2020

Gzip compression doesn't work with DynamoDB (crc32 errors) #526

Gzip compression doesn't work with DynamoDB (crc32 errors) #526

Comments

jakozaur commented Oct 5, 2015

sarxos commented Nov 10, 2015

ohookins commented Dec 22, 2015

zhangzhx commented Dec 22, 2015

zhangzhx commented Dec 22, 2015

jakozaur commented Dec 23, 2015

fernomac commented Dec 23, 2015

fernomac commented Dec 23, 2015

zhangzhx commented Dec 23, 2015

iCesofT commented Mar 31, 2016

bcarl commented Apr 16, 2016

zhangzhx commented Apr 21, 2016

jorgebsa commented May 17, 2016

zhangzhx commented Jul 22, 2016

theRealDrumBum commented Jul 22, 2016

zhangzhx commented Jul 22, 2016

theRealDrumBum commented Jul 22, 2016

zhangzhx commented Jul 23, 2016

breedloj commented Jul 25, 2016

thisismana commented Jul 29, 2016

shorea commented Jul 29, 2016

thisismana commented Jul 30, 2016

thisismana commented Aug 1, 2016

shorea commented Aug 1, 2016

zhangzhx commented Aug 1, 2016

thisismana commented Aug 5, 2016

shorea commented Aug 5, 2016

danvallejo commented Aug 16, 2016

zhangzhx commented Aug 17, 2016

danvallejo commented Aug 17, 2016

shorea commented Aug 17, 2016

danvallejo commented Aug 17, 2016

danvallejo commented Aug 17, 2016

danvallejo commented Aug 17, 2016

zhangzhx commented Aug 17, 2016

danvallejo commented Aug 17, 2016

danvallejo commented Aug 17, 2016

shorea commented Aug 17, 2016

danvallejo commented Aug 17, 2016

shorea commented Aug 17, 2016

shorea commented Aug 17, 2016

shorea commented Aug 17, 2016

shorea commented Aug 18, 2016

theRealDrumBum commented Aug 18, 2016

danvallejo commented Aug 18, 2016

danvallejo commented Aug 19, 2016

shorea commented Aug 19, 2016

danvallejo commented Aug 19, 2016

shorea commented Aug 19, 2016

danvallejo commented Aug 19, 2016