Error message in logs that it cannot read from SQS even thought everything works fine

[naggappan]

Version used: graylog-2.4.5-1.qcow2 (openstack image)

Provided following permissions for a user in aws and used it's auth id / token in graylog aws plugin
Full access: cloudwatch / SNS / SQS
Read only: s3 Kinesis

Now everything works fine and I am able to see the messages in dashboard but still I get the following error message in graylog server log,

ERROR [CloudTrailSubscriber] Could not read messages from SQS. This is most likely a misconfiguration of the plugin. Going into sleep loop and retrying. 2018-06-27_11:37:14.47723 java.lang.RuntimeException: Could not parse SNS notification: { 2018-06-27_11:37:14.47783 "Type" : "Notification", 2018-06-27_11:37:14.47887 "MessageId" : "f5c30c53-b7d9-56d7-9e21-5ff84b546a18", 2018-06-27_11:37:14.47909 "TopicArn" : "arn:aws:sns:us-east-1:XXXXXXXXXXX:cloudtrial-logs-write", 2018-06-27_11:37:14.47944 "Message" : "CloudTrail validation message.", 2018-06-27_11:37:14.47979 "Timestamp" : "2018-06-26T08:33:17.736Z", 2018-06-27_11:37:14.48010 "SignatureVersion" : "1", 2018-06-27_11:37:14.48040 "Signature" : "KXUJMr7gK2Rq5YzLJe2uPHc0Ar2iQaVQREzpVn0+vm+ta3nzcnwxHbHp7dJMxLaYCmW8WCUHy4Oa3mVJIp8aUw93o+yLW211tx+WbH1+GodZDi3hrTz2YyUmvf0nFtcNrql9XNHCiWxs08wKYGhRTd0dCfunU8lusPXfXB5R/nx8s6TDijrxodQdQOLYra2r1eDGFnoCBP5PBdrz7KoNmkuIm67LWzlCx31m9e5cptrqQgTLVirQmalyuZhyADBg3cBoZMRHN7MEtHcvxo3VeBJX4j7BgXd60nYDxlpKoaH1Xw/xLCh+ehZ9dAc76RdjW3xxHA/7PfOUZ+nfjg4Gug==", 2018-06-27_11:37:14.48077 "SigningCertURL" : "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-eaea6120e663d32f88dcd8bcbddca752.pem", 2018-06-27_11:37:14.48114 "UnsubscribeURL" : "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:XXXXXXXXXX:cloudtrial-logs-write:8fe89097-7390-4910-8dc0-021dc808d7af" 2018-06-27_11:37:14.48153 } 2018-06-27_11:37:14.48190 at org.graylog.aws.inputs.cloudtrail.notifications.CloudtrailSNSNotificationParser.parse(CloudtrailSNSNotificationParser.java:43) ~[graylog-plugin-aws-2.4.5.jar:?] 2018-06-27_11:37:14.48229 at org.graylog.aws.inputs.cloudtrail.notifications.CloudtrailSQSClient.getNotifications(CloudtrailSQSClient.java:54) ~[graylog-plugin-aws-2.4.5.jar:?] 2018-06-27_11:37:14.48273 at org.graylog.aws.inputs.cloudtrail.CloudTrailSubscriber.run(CloudTrailSubscriber.java:88) [graylog-plugin-aws-2.4.5.jar:?] 2018-06-27_11:37:14.48382 Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'CloudTrail': was expecting ('true', 'false' or 'null') 2018-06-27_11:37:14.48409 at [Source: CloudTrail validation message.; line: 1, column: 11] 2018-06-27_11:37:14.48449 at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[graylog.jar:?] 2018-06-27_11:37:14.48541 at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[graylog.jar:?] 2018-06-27_11:37:14.48632 at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2839) ~[graylog.jar:?] 2018-06-27_11:37:14.49206 at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1903) ~[graylog.jar:?] 2018-06-27_11:37:14.49291 at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:749) ~[graylog.jar:?] 2018-06-27_11:37:14.49362 at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3850) ~[graylog.jar:?] 2018-06-27_11:37:14.49617 at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3799) ~[graylog.jar:?] 2018-06-27_11:37:14.49677 at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2858) ~[graylog.jar:?] 2018-06-27_11:37:14.49753 at org.graylog.aws.inputs.cloudtrail.notifications.CloudtrailSNSNotificationParser.parse(CloudtrailSNSNotificationParser.java:30) ~[?:?] 2018-06-27_11:37:14.49797 ... 2 more

Seams like the issue is closed: #44
But it works with reporting an error.

Is this a bug or I have done some wrong configuration

[danotorrey]

The error is occurring, because the message payload of the SQS notification contains the text CloudTrail validation message. instead of valid S3 keys (eg.).

{
  "Type": "Notification",
  "MessageId": "f5c30c53-b7d9-56d7-9e21-5ff84b546a18",
  "TopicArn": "arn:aws:sns:us-east-1:XXXXXXXXXXX:cloudtrial-logs-write",
  "Message": "CloudTrail validation message.",
  "Timestamp": "2018-06-26T08:33:17.736Z",
  "SignatureVersion": "1",
  "Signature": "KXUJMr7gK2Rq5YzLJe2uPHc0Ar2iQaVQREzpVn0+vm+ta3nzcnwxHbHp7dJMxLaYCmW8WCUHy4Oa3mVJIp8aUw93o+yLW211tx+WbH1+GodZDi3hrTz2YyUmvf0nFtcNrql9XNHCiWxs08wKYGhRTd0dCfunU8lusPXfXB5R/nx8s6TDijrxodQdQOLYra2r1eDGFnoCBP5PBdrz7KoNmkuIm67LWzlCx31m9e5cptrqQgTLVirQmalyuZhyADBg3cBoZMRHN7MEtHcvxo3VeBJX4j7BgXd60nYDxlpKoaH1Xw/xLCh+ehZ9dAc76RdjW3xxHA/7PfOUZ+nfjg4Gug==",
  "SigningCertURL": "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-eaea6120e663d32f88dcd8bcbddca752.pem",
  "UnsubscribeURL": "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:XXXXXXXXXX:cloudtrial-logs-write:8fe89097-7390-4910-8dc0-021dc808d7af"
}

I have seen this before, and I believe it is some kind of test message. The issue with the Graylog CloudTrail code is that it currently throws a RuntimeException, which halts processing. PR #91 fixes this and just logs the exception and keeps processing. We are expecting this change to be available in release 2.5.

[danotorrey]

Fixes #91 by preventing the AWS Logs/AWS Flow Logs inputs from terminating when they hit an exception (due to a test AWS message). An error will still be logged, but processing will continue for subsequent messages.

[bernd]

Fixed in #91. This will be in the upcoming 2.5 release.