Add AWS CloudTrail Support #15227

leehinman · 2019-12-19T20:16:29Z

Direct Field Mappings

Other mappings

To Do

move aws.cloudtrail.related.users to related.users when ECS is upgraded to 1.4

elasticmachine · 2019-12-20T18:27:46Z

Pinging @elastic/siem (Team:SIEM)

webmat

Most of the mapping to ECS makes sense to me.

I agree with all the fields that are kept as custom fields under aws.* as well. I don't think you've missed any applicable ECS fields for those.

The only thing that's a hard no for me is populating event.category with a value from the source. This is one of the 4 ECS categorization fields, and they're meant to be populated by one of the allowed values defined here. We've just started publishing them in ECS 1.4 (prior to that the field was reserved), and more values are to come. If none of the allowed values for this fields are appropriate for your events (which I think is the case here), the field must be left empty.

webmat · 2019-12-20T19:46:16Z

x-pack/filebeat/module/aws/cloudtrail/ingest/pipeline.yml

+  - rename:
+      field: "json.eventType"
+      target_field: event.category
+      ignore_failure: true


event.category should never be populated with a source value.

In ECS 1.4, we're publishing the first allowed values for this field and other (see this new section).

If none of the values published so far for this field (more to come), this event.category field must be left empty.

So you should capture json.eventType in a custom field.

Thanks for making that adjustment. LGTM from ECS' POV :-)

You should likely get approval from another Beats maintainer as well, IMO. I only reviewed the mapping of the events.

Thanks for reviewing, that was a big help and gives me a little more confidence that I'm using ECS correctly.

It will be a bit before it is ready to merge, I still have to work out the S3 bucket parts.

webmat

Thanks for making that adjustment. LGTM from ECS' POV :-)

You should likely get approval from another Beats maintainer as well, IMO. I only reviewed the mapping of the events.

kaiyan-sheng · 2019-12-23T21:57:22Z

x-pack/filebeat/module/aws/cloudtrail/_meta/fields.yml

@@ -0,0 +1,104 @@
+- name: cloudtrail
+  type: group


Could you please add release: beta here please? I forgot to add it for previous filesets so I just created a PR to add it: https://github.com/elastic/beats/pull/15248/files

Added release field. Thanks for catching this

andrewkroh · 2020-01-02T19:32:58Z

x-pack/filebeat/module/aws/cloudtrail/test/update-trail-json.log-expected.json

+        "source.address": "205.251.233.182",
+        "user.id": "EX_PRINCIPAL_ID",
+        "user.name": "Alice",
+        "user_agent.original": "aws-cli/1.10.32 Python/2.7.9 Windows/7 botocore/1.4.22"


You can apply the user_agent processor to the Ingest Node configuration to parse the value.

andrewkroh · 2020-01-02T19:33:50Z

x-pack/filebeat/module/aws/cloudtrail/test/update-trail-json.log-expected.json

+        "input.type": "log",
+        "log.offset": 0,
+        "service.type": "aws",
+        "source.address": "205.251.233.182",


This should get copied over to the source.ip field when its an IP.

Do we have a good test for an IP address(4 & 6)? The Cloudtrail docs say "For services in AWS, only the DNS name is displayed". So we have to tell the difference, seeing if geoip works almost does it, but geoip doesn't work for localhost and private IP space.

andrewkroh · 2020-01-02T19:37:32Z

filebeat/docs/fields.asciidoc

+--
+The date and time when the temporary security credentials were issued.
+
+type: keyword


Perhaps this should be an actual date type?

yes it should

ynirk · 2020-01-06T15:42:55Z

I have tested a configuration on filebeat 7.5.1 based on this PR.
I have a parsing issue because cloudtrail events are stored in a Records array and an S3 object might contain several events. (some exemples are shown in cloudtrail logs exemples)
I don't know if there is an easy way to split records array elements to single beats events (like logstash split filter)

kaiyan-sheng · 2020-01-07T15:42:18Z

I just created #15357 to track the work on adding json support for S3 input.

kaiyan-sheng · 2020-01-13T14:54:43Z

I hit an error when running this fileset:

'cloudtrail.amazonaws.com' is not an IP string literal.

Original message:

{"additionalEventData":{"AuthenticationMethod":"AuthHeader","CipherSuite":"ECDHE-RSA-AES128-SHA","SSEApplied":"SSE_S3","SignatureVersion":"SigV4","bytesTransferredIn":758,"bytesTransferredOut":0,"x-amz-id-2":"4LMgpirqOBiwClBqCrOg3Z/53SBdgrMR5+n9ZyJd0xY5bFL/nMFGW3VqF+6rJ4FO68GurEbQSLI="},"awsRegion":"us-east-1","eventID":"312ee59e-59eb-4056-8193-704368f1712d","eventName":"PutObject","eventSource":"s3.amazonaws.com","eventTime":"2020-01-10T17:01:03Z","eventType":"AwsApiCall","eventVersion":"1.05","readOnly":false,"recipientAccountId":"428152502467","requestID":"3D058BA57D0782F3","requestParameters":{"Host":"test-fb-ks.s3.amazonaws.com","bucketName":"test-fb-ks","key":"AWSLogs/428152502467/CloudTrail/us-east-1/2020/01/10/428152502467_CloudTrail_us-east-1_20200110T1700Z_dRSvtwdbSGr4vvAE.json.gz","x-amz-acl":"bucket-owner-full-control","x-amz-server-side-encryption":"AES256"},"resources":[{"ARN":"arn:aws:s3:::test-fb-ks/AWSLogs/428152502467/CloudTrail/us-east-1/2020/01/10/428152502467_CloudTrail_us-east-1_20200110T1700Z_dRSvtwdbSGr4vvAE.json.gz","type":"AWS::S3::Object"},{"ARN":"arn:aws:s3:::test-fb-ks","accountId":"428152502467","type":"AWS::S3::Bucket"}],"responseElements":{"x-amz-server-side-encryption":"AES256"},"sharedEventID":"c9503de9-f712-423c-a8fe-7fe2bc2fe9bf","sourceIPAddress":"cloudtrail.amazonaws.com","userAgent":"cloudtrail.amazonaws.com","userIdentity":{"invokedBy":"cloudtrail.amazonaws.com","type":"AWSService"}}

kaiyan-sheng

Also could you add this fileset does not support CloudTrail-Digest logs in the documentation please? Thanks!

leehinman · 2020-01-13T15:02:44Z

{"additionalEventData":{"AuthenticationMethod":"AuthHeader","CipherSuite":"ECDHE-RSA-AES128-SHA","SSEApplied":"SSE_S3","SignatureVersion":"SigV4","bytesTransferredIn":758,"bytesTransferredOut":0,"x-amz-id-2":"4LMgpirqOBiwClBqCrOg3Z/53SBdgrMR5+n9ZyJd0xY5bFL/nMFGW3VqF+6rJ4FO68GurEbQSLI="},"awsRegion":"us-east-1","eventID":"312ee59e-59eb-4056-8193-704368f1712d","eventName":"PutObject","eventSource":"s3.amazonaws.com","eventTime":"2020-01-10T17:01:03Z","eventType":"AwsApiCall","eventVersion":"1.05","readOnly":false,"recipientAccountId":"428152502467","requestID":"3D058BA57D0782F3","requestParameters":{"Host":"test-fb-ks.s3.amazonaws.com","bucketName":"test-fb-ks","key":"AWSLogs/428152502467/CloudTrail/us-east-1/2020/01/10/428152502467_CloudTrail_us-east-1_20200110T1700Z_dRSvtwdbSGr4vvAE.json.gz","x-amz-acl":"bucket-owner-full-control","x-amz-server-side-encryption":"AES256"},"resources":[{"ARN":"arn:aws:s3:::test-fb-ks/AWSLogs/428152502467/CloudTrail/us-east-1/2020/01/10/428152502467_CloudTrail_us-east-1_20200110T1700Z_dRSvtwdbSGr4vvAE.json.gz","type":"AWS::S3::Object"},{"ARN":"arn:aws:s3:::test-fb-ks","accountId":"428152502467","type":"AWS::S3::Bucket"}],"responseElements":{"x-amz-server-side-encryption":"AES256"},"sharedEventID":"c9503de9-f712-423c-a8fe-7fe2bc2fe9bf","sourceIPAddress":"cloudtrail.amazonaws.com","userAgent":"cloudtrail.amazonaws.com","userIdentity":{"invokedBy":"cloudtrail.amazonaws.com","type":"AWSService"}}

ick, that is because sourcIPAddress contains a hostname and not an IP address, so the geoip processor doesn't like it. Thanks.

leehinman · 2020-01-13T16:44:26Z

Also could you add this fileset does not support CloudTrail-Digest logs in the documentation please? Thanks!

Added that to asciidoc & readme. thanks.

- maps all fields in CloudTrail events - requestParameters, responseElements, additionalEventData & serviceEventDetails are string representations - add event.original - add event.type - add event.kind - add event.outcome - run geoip processor - run agent processor - populated related.user array when possible - uses s3input - CloudTrail must write to S3 bucket, and send all Create Events to an SQS queue we listen to Fixes elastic#14657

- maps all fields in CloudTrail events - requestParameters, responseElements, additionalEventData & serviceEventDetails are string representations - add event.original - add event.type - add event.kind - add event.outcome - run geoip processor - run agent processor - populated related.user array when possible - uses s3input - CloudTrail must write to S3 bucket, and send all Create Events to an SQS queue we listen to Fixes elastic#14657 (cherry picked from commit da7a697)

- maps all fields in CloudTrail events - requestParameters, responseElements, additionalEventData & serviceEventDetails are string representations - add event.original - add event.type - add event.kind - add event.outcome - run geoip processor - run agent processor - populated related.user array when possible - uses s3input - CloudTrail must write to S3 bucket, and send all Create Events to an SQS queue we listen to Fixes #14657 (cherry picked from commit da7a697)

leehinman added the Filebeat Filebeat label Dec 19, 2019

leehinman requested review from webmat, cwurm, andrewkroh and MikePaquette December 19, 2019 20:16

andrewkroh added the Team:SIEM label Dec 20, 2019

andrewkroh mentioned this pull request Dec 20, 2019

[Filebeat] Add fileset for AWS CloudTrail #14657

Closed

webmat suggested changes Dec 20, 2019

View reviewed changes

webmat approved these changes Dec 20, 2019

View reviewed changes

kaiyan-sheng reviewed Dec 23, 2019

View reviewed changes

leehinman force-pushed the 14657_aws_cloudtrail branch from b6d30ac to f9e7419 Compare December 23, 2019 22:29

andrewkroh reviewed Jan 2, 2020

View reviewed changes

leehinman force-pushed the 14657_aws_cloudtrail branch 3 times, most recently from 93d64bb to 98a2494 Compare January 11, 2020 00:45

leehinman marked this pull request as ready for review January 13, 2020 14:35

leehinman requested a review from a team as a code owner January 13, 2020 14:35

leehinman added the needs_backport PR is waiting to be backported to other branches. label Jan 13, 2020

kaiyan-sheng reviewed Jan 13, 2020

View reviewed changes

leehinman force-pushed the 14657_aws_cloudtrail branch 2 times, most recently from ecf5ef4 to 398ffa7 Compare January 13, 2020 17:50

kaiyan-sheng approved these changes Jan 13, 2020

View reviewed changes

leehinman force-pushed the 14657_aws_cloudtrail branch from 398ffa7 to 0bb68e1 Compare January 13, 2020 20:25

andrewkroh approved these changes Jan 13, 2020

View reviewed changes

leehinman merged commit da7a697 into elastic:master Jan 13, 2020

leehinman mentioned this pull request Jan 13, 2020

Cherry-pick #15227 to 7.x: Add AWS CloudTrail Support #15519

Merged

39 tasks

leehinman added v7.6.0 and removed needs_backport PR is waiting to be backported to other branches. labels Jan 13, 2020

leehinman deleted the 14657_aws_cloudtrail branch January 21, 2020 20:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add AWS CloudTrail Support #15227

Add AWS CloudTrail Support #15227

leehinman commented Dec 19, 2019 •

edited

elasticmachine commented Dec 20, 2019

webmat left a comment •

edited

webmat Dec 20, 2019

leehinman Dec 20, 2019

webmat left a comment

kaiyan-sheng Dec 23, 2019

leehinman Dec 23, 2019

andrewkroh Jan 2, 2020

leehinman Jan 7, 2020

andrewkroh Jan 2, 2020

leehinman Jan 7, 2020

andrewkroh Jan 2, 2020

leehinman Jan 7, 2020

ynirk commented Jan 6, 2020

kaiyan-sheng commented Jan 7, 2020

kaiyan-sheng commented Jan 13, 2020

kaiyan-sheng left a comment

leehinman commented Jan 13, 2020

leehinman commented Jan 13, 2020

Add AWS CloudTrail Support #15227

Add AWS CloudTrail Support #15227

Conversation

leehinman commented Dec 19, 2019 • edited

Direct Field Mappings

Other mappings

To Do

elasticmachine commented Dec 20, 2019

webmat left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ynirk commented Jan 6, 2020

kaiyan-sheng commented Jan 7, 2020

kaiyan-sheng commented Jan 13, 2020

kaiyan-sheng left a comment

Choose a reason for hiding this comment

leehinman commented Jan 13, 2020

leehinman commented Jan 13, 2020

leehinman commented Dec 19, 2019 •

edited

webmat left a comment •

edited