MD5 Mismatch for some SQS messages #13

Closed
vdex42 opened this Issue Dec 10, 2013 · 11 comments

Comments

Projects
None yet
5 participants

vdex42 commented Dec 10, 2013

Firstly, thank you for making this excellent tool, it has been very useful.

I have found that some SQS messages when read through the Amazon SDK result in errors like:
MD5 hash mismatch for message id d97a1912-c5a3-4f41-8aa5-e6e37a87474f
at Amazon.SQS.Util.AmazonSQSUtil.ValidateMD5(String message, String messageId, String md5FromService)
at Amazon.SQS.Util.AmazonSQSUtil.ValidateMD5(Message message)
at Amazon.SQS.Util.AmazonSQSUtil.ValidateReceiveMessage(ReceiveMessageResponse response)
at Amazon.SQS.AmazonSQSClient.ReceiveMessage(ReceiveMessageRequest request)

I have experimented with the text I am sending. The following test case will result in that error. (It is worth adding that this message does work on the real amazon SQS)
"{"Body":"{\"Event\":\"{\\"RequestId\\":\\"b6889680-c986-4f99-9dff-a29000a800b1\\"\\"EmailSubject\\":\\"Toolbox – reset password\\"}\"}"}"

However other more complex messages go through fine.
Thank you,
Ivan

Owner

adamw commented Dec 12, 2013

Thanks for the report, this looks quite weird, I'll take a look in a while :)

wikyd commented Sep 11, 2014

Agreed, thanks for building this! Makes testing locally much easier. I actually ran into this same issue and here was how I resolved it.

I was sending messages using the Java AWS SDK. My messages looked kind of weird when sent as a String because I was actually just sending a byte[] wrapped as a String. It turns out that there are some invalid characters that SQS does not accept (perhaps the mdash in the email subject in this example). I'm not sure exactly what is happening, but I think the MD5 calculation is different because one message body is sanitized and the other version is not.

I fixed it by just Url Safe base64 encoding first.

Owner

adamw commented Sep 16, 2014

Unfortunately I can't reproduce this locally. There are indeed forbidden characters in SQS, but such messages should be rejected and are rejected by both SQS and ElasticMQ. I tried the message body from above but works fine with the latest version.

Please reopen if anybody encounters this problem again.

adamw closed this Sep 16, 2014

I'm getting this problem now.

I had some integration tests which were all passing. I had a requirement to add a couple of new attributes to the message.

I ran the tests. They passed. I added the 2 new attributes. Then I got:

Attribute MD5 hash mismatch

I then removed the 2 attributes and goot:

Attribute MD5 hash mismatch for message id 760349d3-0d7f-4593-bfa0-1a15cb0c6ca3

The queue is now fucked. I literally added 2 attributes. Then removed them. Now I can't do anything.

When adding the message attributes, we have mixed casing on the names. For example:

image

As you can see here, ftpuser and ftpaddress are lowercase. While the other two are uppercase.

If I change the two values to be FtpUser and FtpAddress my error goes away.

It's like there is inconsistency when doing the the MD5 hash but I can only find the validation one, I cannot find the send request one to compare what they are doing.

https://github.com/aws/aws-sdk-net/blob/master/AWSSDK_DotNet35/Amazon.SQS/Internal/ValidationResponseHandler.cs#L143

Can confirm, you have a casing bug in your SDK.

If the attribute names are all uppercase, or all lowercase. There's 0 issues.

If you have some upper some lower it fails 100% of the time.

You can read not but write, or not write, depending on the order you attempt to do this stuff in.

Owner

adamw commented Mar 6, 2015

So it's a bug somewhere in ElasticMQ you say? :)

@adamw I'm not sure if it's related to ElasticMQ, we are using SQS. I didn't realise this was logged under ElasticMQ because this was like the only link that popped up with the error message.

So it's a bug with SQS, not system.

Owner

adamw commented Mar 6, 2015

Ah ok, thanks for the clarification :)

Contributor

kopchik commented Feb 7, 2017

Hi @adamw,

I'm experiencing the same issue. I'm no expert in Java/Scala, but might the issue is caused by data conversion? E.g., unicode symbols can be lost if system locale is not unicode and getBytes() is not provided which encoding to use.
So, please consider this patch: https://gist.github.com/kopchik/173e9c3caa99c0a787a90d0e2da85ca6 .

Owner

adamw commented Feb 7, 2017 edited

@kopchik Ah! nice catch, could be :) Could you maybe convert the gist into a PR? that way you'll be properly included in the history :)

@kopchik kopchik added a commit to kopchik/elasticmq that referenced this issue Feb 7, 2017

@kopchik kopchik md5Digest(): explicitly provide getBytes() encoding
If encoding not provided, String.getBytes() uses system encoding to convert string into array of bytes. If system encoding is not set or it is not unicode, then unicode symbols may be lost/altered (we experienced this in our docker container). As a result, md5 checksum will be wrong (I'm not sure if the message itself gets corrupted). This should fix #13.
ecdc364
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment