SNS - SES Notification - Invalid signature if special characters are present

[jplippi]

When receiving SES notifications through an HTTP/S endpoint subscribed to a SNS topic, the SDK cannot validate the signature if the string to sign contains special characters, such as á or ç

Describe the bug

Signature verification fails when the string to sign resulting of the notification contains special characters. Since the string signature contains the email subject, the subject is susceptible to contain special characters

Expected Behavior

Signature validation passes

Current Behavior

Signature validation fails

Steps to Reproduce

• Create a SNS topic and subscribe to a Java web application endpoint
• Configure SES to have a Configuration Set with an SNS destination and Bounce event type enabled
• Send an email through SES SDK with the pre-configured Configuration Set, and with special characters in the Subject field
• Use SNS SDK to handle the message:
  SnsMessageManager().handleMessage(inputStream, SnsMessageHandler())
• Verify that SnsMessageHandler will throw an exception SdkClientException("Signature in SNS message was invalid")

Context

When trying to verify the signature of the SNS message to handle email bounces and complaints, signature validation would fail only for emails that contained special characters on the subject field

Your Environment

• AWS Java SDK version used: 1.11.791
• JDK version used: 1.8
• Operating System and version: Windows 10 Pro 64-bit (10.0, Build 18362)

[debora-ito]

@jplippi what version of Apache httpclient are you using?

We had issues with httpclient changing its behavior in version 4.5.7 and affecting special characters (see #1919), just want to eliminate this possibility before doing a deep dive in the problem.

[jplippi]

@debora-ito httpclient version 4.5. I just did some testing with 4.5.12 as well, and had the same problem

[debora-ito]

Can you provide a code sample that I can use to reproduce?

[jplippi]

@debora-ito I've made a simple Tomcat Web Application, since this is the usecase that I currently have:
https://github.com/jplippi/aws-sns-bug

I hope this helps!

[jplippi]

Ok, I made a simple unit test method that didn't reproduce the issue. The validation was successful through a unit test, but not with an HTTP request to the web application.

I put a breakpoint inside SignatureChecker.verifySignature() and even though the String content for the parameter message is the same, the method message.getBytes() returns two different sized byte arrays in each case.

When using the unit test, I got a byte[1570], but when using the API endpoint, got a byte[1560].

I noticed that if I specify a charset whtn using message.getBytes() I can reproduce the behavior:

message.getBytes(Charsets.ISO_8859_1) returns a byte[1560]
message.getBytes(Charsets.UTF_8) returns a byte[1570]

So I believe an explicit Charset has to be provided for the signature checker.

I don't know if it falls within the scope of the SDK or if I should handle the encoding myself before sending the InputStream

[jplippi]

Ok, I verified it to be the case:
When running an unit test, Charset.defaultCharset() returns "UTF-8", but when running the application from Tomcat, Charset.defaultCharset() returns "windows-1252".

Since a specific encoding is required for the signature verifiction, I believe it has to be explicitly set within the SDK

[debora-ito]

Quick update: I'm still working on the repro case set up, will work on the issue this week.

[debora-ito]

@jplippi apologies for losing track of this.

Since you identified the issue was in the encoding of the signature checker, at this point I'll recommend you add the encoding to your application. If we add the encoding to the SDK, there's a risk we'll be double-encoding the content for customers who already added the encoding to their code, and this will be a breaking change.