S3 bucket access logs?

[drummerjoe]

This is not an issue but a question!

I've got this working perfectly for CloudTrail logs - it's great! I'm trying to find a way to ship S3 bucket access logs using it - I've tried a combo of parseSpaces with the CloudFront type and no unzipping, and this does shove the data into ES, but there's no handling of the format to speak of. I've tried forcing it through an index template in ES but that's also pretty messy.

Is there an optimal way of shipping S3 access logs using lambda-stash or am I out of luck?

[arithmetric]

Good question, @drummerjoe.

I found documentation about the S3 server access logs format here: https://docs.aws.amazon.com/AmazonS3/latest/dev/LogFormat.html

Since this format is different than the other supported formats, parsing this file into structured data will need a new handler. This format is similar to the ELB access logs in that it is space delimited and does not include any headers in the file, so the handler can be modeled on formatELBv2.js.

If you're able to implement this, I'd welcome it as a contribution.

[drummerjoe]

I'm on it :)

[drummerjoe]

Does item[config.dateField] require any particular format pre-shipping to ES? I can't quite get ES to pick this up as a date field unless I force it through an index template which I'd prefer not to do.

[arithmetric]

@drummerjoe, I see that logs for some of the other AWS services supported by this script use ISO-8601 format for dates (like 2016-06-16T14:19:13Z). This script is able to pass those values directly to ES.

The S3 access logs use a different format. An example is: 06/Feb/2019:00:00:38 +0000

I think it's appropriate for this handler to reformat the date value to ISO-8601 so that no handling is needed on the ES side.

[drummerjoe]

@arithmetric, that was it - once I reformatted the date ES took it without complaint. Works exactly as I'd hoped. I'll clean it up and send you a PR.