Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Iterate over log events & post them individually #7

Merged
merged 1 commit into from Apr 6, 2016

Conversation

dgarlitt
Copy link
Contributor

CloudWatch often bundles multiple log events together in the event input
archive that is passed in the lambda's event argument. Since the lambda
was only doing a single post to the collection, that post would in many
cases actually contain multiple logs which would then show up as a
single concatenated log in the collection resulting in incorrect metrics.

This change iterates over the log events in the archive and posts them
individually to the collection.

@duchatran
Copy link
Contributor

Thanks @dgarlitt! I notice the code to extract and include RequestID as a separate field in the JSON to Sumo was removed. Since this was actually requested by some customer, it would be good to have it back. I am thinking about making it optional if not everyone needs it.

@dgarlitt
Copy link
Contributor Author

@duchatran Good catch, it's updated to include that logic.

@dgarlitt dgarlitt force-pushed the iterate-over-log-events branch 2 times, most recently from de64c74 to 3be0966 Compare March 30, 2016 16:51
@dgarlitt
Copy link
Contributor Author

@duchatran I just updated this commit again to adjust the formatting on the options object at the top so that they are all in line. It was strictly cosmetic. Let me know if there are any other concerns about this pull request.

@duchatran
Copy link
Contributor

Thanks @dgarlitt. Sorry for not getting back sooner, was caught up with something else. Let me do some test and then I will merge this.

@dgarlitt
Copy link
Contributor Author

dgarlitt commented Apr 4, 2016

👍 Thanks!

var tot = requestsSent + requestsFailed;
if (tot == awslogsData.logEvents.length) {
if (requestsFailed > 0) {
context.fail();
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

context.fail and context.succeed allow to pass in a string value. Changing your code to this might be a nice optional addition:

if (requestsFailed > 0) {
  context.fail(requestsFailed + " / " + (requestsFailed + requestsSent) + " events failed");
} else {
  context.succeed(requestsSent + " requests sent");
}

@dgarlitt dgarlitt force-pushed the iterate-over-log-events branch 2 times, most recently from d8dbb07 to 44f5b44 Compare April 5, 2016 21:39
@dgarlitt
Copy link
Contributor Author

dgarlitt commented Apr 5, 2016

@thoean I amended the commit to include the change regarding your suggestion on adding info to context.fail(). Github is having issues right now, so it may be a bit before the "Checking for ability to automatically merge..." status goes away.

@dgarlitt dgarlitt force-pushed the iterate-over-log-events branch 2 times, most recently from d8dbb07 to de457d4 Compare April 5, 2016 21:53
CloudWatch often bundles multiple log events together in the event input
archive that is passed in the lambda's event argument. Since the lambda
was only doing a single post to the collection, that post would in many
cases actually contain multiple logs which would then show up as a
single concatenated log in the collection resulting in incorrect metrics.

This change iterates over the log events in the archive and posts them
individually to the collection.
@dgarlitt
Copy link
Contributor Author

dgarlitt commented Apr 5, 2016

@thoean Added info to context.succeed() as well. Good suggestion, btw.

@duchatran duchatran merged commit ade3610 into SumoLogic:master Apr 6, 2016
@dgarlitt dgarlitt deleted the iterate-over-log-events branch April 25, 2016 16:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants