MD5 Mismatch for some SQS messages

[vdex42]

Firstly, thank you for making this excellent tool, it has been very useful.

I have found that some SQS messages when read through the Amazon SDK result in errors like:
MD5 hash mismatch for message id d97a1912-c5a3-4f41-8aa5-e6e37a87474f
at Amazon.SQS.Util.AmazonSQSUtil.ValidateMD5(String message, String messageId, String md5FromService)
at Amazon.SQS.Util.AmazonSQSUtil.ValidateMD5(Message message)
at Amazon.SQS.Util.AmazonSQSUtil.ValidateReceiveMessage(ReceiveMessageResponse response)
at Amazon.SQS.AmazonSQSClient.ReceiveMessage(ReceiveMessageRequest request)

I have experimented with the text I am sending. The following test case will result in that error. (It is worth adding that this message does work on the real amazon SQS)
"{"Body":"{\"Event\":\"{\\"RequestId\\":\\"b6889680-c986-4f99-9dff-a29000a800b1\\"\\"EmailSubject\\":\\"Toolbox – reset password\\"}\"}"}"

However other more complex messages go through fine.
Thank you,
Ivan

[adamw]

Thanks for the report, this looks quite weird, I'll take a look in a while :)

[wikyd]

Agreed, thanks for building this! Makes testing locally much easier. I actually ran into this same issue and here was how I resolved it.

I was sending messages using the Java AWS SDK. My messages looked kind of weird when sent as a String because I was actually just sending a byte[] wrapped as a String. It turns out that there are some invalid characters that SQS does not accept (perhaps the mdash in the email subject in this example). I'm not sure exactly what is happening, but I think the MD5 calculation is different because one message body is sanitized and the other version is not.

I fixed it by just Url Safe base64 encoding first.

[adamw]

Unfortunately I can't reproduce this locally. There are indeed forbidden characters in SQS, but such messages should be rejected and are rejected by both SQS and ElasticMQ. I tried the message body from above but works fine with the latest version.

Please reopen if anybody encounters this problem again.

[phillip-haydon]

I'm getting this problem now.

I had some integration tests which were all passing. I had a requirement to add a couple of new attributes to the message.

I ran the tests. They passed. I added the 2 new attributes. Then I got:

Attribute MD5 hash mismatch

I then removed the 2 attributes and goot:

Attribute MD5 hash mismatch for message id 760349d3-0d7f-4593-bfa0-1a15cb0c6ca3

The queue is now fucked. I literally added 2 attributes. Then removed them. Now I can't do anything.

[phillip-haydon]

When adding the message attributes, we have mixed casing on the names. For example:

As you can see here, ftpuser and ftpaddress are lowercase. While the other two are uppercase.

If I change the two values to be FtpUser and FtpAddress my error goes away.

It's like there is inconsistency when doing the the MD5 hash but I can only find the validation one, I cannot find the send request one to compare what they are doing.

https://github.com/aws/aws-sdk-net/blob/master/AWSSDK_DotNet35/Amazon.SQS/Internal/ValidationResponseHandler.cs#L143

[phillip-haydon]

Can confirm, you have a casing bug in your SDK.

If the attribute names are all uppercase, or all lowercase. There's 0 issues.

If you have some upper some lower it fails 100% of the time.

You can read not but write, or not write, depending on the order you attempt to do this stuff in.

[adamw]

So it's a bug somewhere in ElasticMQ you say? :)

[phillip-haydon]

@adamw I'm not sure if it's related to ElasticMQ, we are using SQS. I didn't realise this was logged under ElasticMQ because this was like the only link that popped up with the error message.

So it's a bug with SQS, not system.

[adamw]

Ah ok, thanks for the clarification :)

[kopchik]

Hi @adamw,

I'm experiencing the same issue. I'm no expert in Java/Scala, but might the issue is caused by data conversion? E.g., unicode symbols can be lost if system locale is not unicode and getBytes() is not provided which encoding to use.
So, please consider this patch: https://gist.github.com/kopchik/173e9c3caa99c0a787a90d0e2da85ca6 .

[adamw]

@kopchik Ah! nice catch, could be :) Could you maybe convert the gist into a PR? that way you'll be properly included in the history :)