New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error message in logs that it cannot read from SQS even thought everything works fine #80

Closed
naggappan opened this Issue Jun 27, 2018 · 3 comments

Comments

Projects
None yet
4 participants
@naggappan

naggappan commented Jun 27, 2018

Version used: graylog-2.4.5-1.qcow2 (openstack image)

Provided following permissions for a user in aws and used it's auth id / token in graylog aws plugin
Full access: cloudwatch / SNS / SQS
Read only: s3 Kinesis

Now everything works fine and I am able to see the messages in dashboard but still I get the following error message in graylog server log,

ERROR [CloudTrailSubscriber] Could not read messages from SQS. This is most likely a misconfiguration of the plugin. Going into sleep loop and retrying. 2018-06-27_11:37:14.47723 java.lang.RuntimeException: Could not parse SNS notification: { 2018-06-27_11:37:14.47783 "Type" : "Notification", 2018-06-27_11:37:14.47887 "MessageId" : "f5c30c53-b7d9-56d7-9e21-5ff84b546a18", 2018-06-27_11:37:14.47909 "TopicArn" : "arn:aws:sns:us-east-1:XXXXXXXXXXX:cloudtrial-logs-write", 2018-06-27_11:37:14.47944 "Message" : "CloudTrail validation message.", 2018-06-27_11:37:14.47979 "Timestamp" : "2018-06-26T08:33:17.736Z", 2018-06-27_11:37:14.48010 "SignatureVersion" : "1", 2018-06-27_11:37:14.48040 "Signature" : "KXUJMr7gK2Rq5YzLJe2uPHc0Ar2iQaVQREzpVn0+vm+ta3nzcnwxHbHp7dJMxLaYCmW8WCUHy4Oa3mVJIp8aUw93o+yLW211tx+WbH1+GodZDi3hrTz2YyUmvf0nFtcNrql9XNHCiWxs08wKYGhRTd0dCfunU8lusPXfXB5R/nx8s6TDijrxodQdQOLYra2r1eDGFnoCBP5PBdrz7KoNmkuIm67LWzlCx31m9e5cptrqQgTLVirQmalyuZhyADBg3cBoZMRHN7MEtHcvxo3VeBJX4j7BgXd60nYDxlpKoaH1Xw/xLCh+ehZ9dAc76RdjW3xxHA/7PfOUZ+nfjg4Gug==", 2018-06-27_11:37:14.48077 "SigningCertURL" : "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-eaea6120e663d32f88dcd8bcbddca752.pem", 2018-06-27_11:37:14.48114 "UnsubscribeURL" : "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:XXXXXXXXXX:cloudtrial-logs-write:8fe89097-7390-4910-8dc0-021dc808d7af" 2018-06-27_11:37:14.48153 } 2018-06-27_11:37:14.48190 at org.graylog.aws.inputs.cloudtrail.notifications.CloudtrailSNSNotificationParser.parse(CloudtrailSNSNotificationParser.java:43) ~[graylog-plugin-aws-2.4.5.jar:?] 2018-06-27_11:37:14.48229 at org.graylog.aws.inputs.cloudtrail.notifications.CloudtrailSQSClient.getNotifications(CloudtrailSQSClient.java:54) ~[graylog-plugin-aws-2.4.5.jar:?] 2018-06-27_11:37:14.48273 at org.graylog.aws.inputs.cloudtrail.CloudTrailSubscriber.run(CloudTrailSubscriber.java:88) [graylog-plugin-aws-2.4.5.jar:?] 2018-06-27_11:37:14.48382 Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'CloudTrail': was expecting ('true', 'false' or 'null') 2018-06-27_11:37:14.48409 at [Source: CloudTrail validation message.; line: 1, column: 11] 2018-06-27_11:37:14.48449 at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[graylog.jar:?] 2018-06-27_11:37:14.48541 at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[graylog.jar:?] 2018-06-27_11:37:14.48632 at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2839) ~[graylog.jar:?] 2018-06-27_11:37:14.49206 at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1903) ~[graylog.jar:?] 2018-06-27_11:37:14.49291 at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:749) ~[graylog.jar:?] 2018-06-27_11:37:14.49362 at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3850) ~[graylog.jar:?] 2018-06-27_11:37:14.49617 at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3799) ~[graylog.jar:?] 2018-06-27_11:37:14.49677 at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2858) ~[graylog.jar:?] 2018-06-27_11:37:14.49753 at org.graylog.aws.inputs.cloudtrail.notifications.CloudtrailSNSNotificationParser.parse(CloudtrailSNSNotificationParser.java:30) ~[?:?] 2018-06-27_11:37:14.49797 ... 2 more

Seams like the issue is closed: #44
But it works with reporting an error.

Is this a bug or I have done some wrong configuration

@danotorrey

This comment has been minimized.

Contributor

danotorrey commented Nov 7, 2018

The error is occurring, because the message payload of the SQS notification contains the text CloudTrail validation message. instead of valid S3 keys (eg.).

{
  "Type": "Notification",
  "MessageId": "f5c30c53-b7d9-56d7-9e21-5ff84b546a18",
  "TopicArn": "arn:aws:sns:us-east-1:XXXXXXXXXXX:cloudtrial-logs-write",
  "Message": "CloudTrail validation message.",
  "Timestamp": "2018-06-26T08:33:17.736Z",
  "SignatureVersion": "1",
  "Signature": "KXUJMr7gK2Rq5YzLJe2uPHc0Ar2iQaVQREzpVn0+vm+ta3nzcnwxHbHp7dJMxLaYCmW8WCUHy4Oa3mVJIp8aUw93o+yLW211tx+WbH1+GodZDi3hrTz2YyUmvf0nFtcNrql9XNHCiWxs08wKYGhRTd0dCfunU8lusPXfXB5R/nx8s6TDijrxodQdQOLYra2r1eDGFnoCBP5PBdrz7KoNmkuIm67LWzlCx31m9e5cptrqQgTLVirQmalyuZhyADBg3cBoZMRHN7MEtHcvxo3VeBJX4j7BgXd60nYDxlpKoaH1Xw/xLCh+ehZ9dAc76RdjW3xxHA/7PfOUZ+nfjg4Gug==",
  "SigningCertURL": "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-eaea6120e663d32f88dcd8bcbddca752.pem",
  "UnsubscribeURL": "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:XXXXXXXXXX:cloudtrial-logs-write:8fe89097-7390-4910-8dc0-021dc808d7af"
}

I have seen this before, and I believe it is some kind of test message. The issue with the Graylog CloudTrail code is that it currently throws a RuntimeException, which halts processing. PR #91 fixes this and just logs the exception and keeps processing. We are expecting this change to be available in release 2.5.

@danotorrey danotorrey added this to the 2.5.0 milestone Nov 7, 2018

@danotorrey danotorrey added known issue and removed to-verify labels Nov 7, 2018

@danotorrey

This comment has been minimized.

Contributor

danotorrey commented Nov 19, 2018

Fixes #91 by preventing the AWS Logs/AWS Flow Logs inputs from terminating when they hit an exception (due to a test AWS message). An error will still be logged, but processing will continue for subsequent messages.

@bernd

This comment has been minimized.

Member

bernd commented Nov 22, 2018

Fixed in #91. This will be in the upcoming 2.5 release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment