[Filebeat] Add AWS CloudTrail Support (elastic#15227)

- maps all fields in CloudTrail events
- requestParameters, responseElements, additionalEventData
  & serviceEventDetails are string representations
- add event.original
- add event.type
- add event.kind
- add event.outcome
- run geoip processor
- run agent processor
- populated related.user array when possible
- uses s3input
- CloudTrail must write to S3 bucket, and send all Create Events
  to an SQS queue we listen to

Fixes elastic#14657

CHANGELOG.next.asciidoc

@@ -486,6 +486,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Include log.source.address for unparseable syslog messages. {issue}13268[13268] {pull}15453[15453]
 - Release aws elb fileset as GA. {pull}15426[15426] {issue}15380[15380]
 - Release aws s3access fileset to GA. {pull}15431[15431] {issue}15430[15430]
+- Add cloudtrail fileset to AWS module. {issue}14657[14657] {pull}15227[15227]
 
 *Heartbeat*
 - Add non-privileged icmp on linux and darwin(mac). {pull}13795[13795] {issue}11498[11498]

filebeat/docs/fields.asciidoc

@@ -1070,6 +1070,249 @@ Fields from AWS logs.
 
 
 
+[float]
+=== cloudtrail
+
+Fields for AWS CloudTrail logs.
+
+
+
+*`aws.cloudtrail.event_version`*::
++
+--
+The CloudTrail version of the log event format.
+
+
+type: keyword
+
+--
+
+[float]
+=== user_identity
+
+The userIdentity element contains details about the type of IAM identity that made the request, and which credentials were used. If temporary credentials were used, the element shows how the credentials were obtained.
+
+
+*`aws.cloudtrail.user_identity.type`*::
++
+--
+The type of the identity
+
+
+type: keyword
+
+--
+
+*`aws.cloudtrail.user_identity.arn`*::
++
+--
+The Amazon Resource Name (ARN) of the principal that made the call.
+
+type: keyword
+
+--
+
+*`aws.cloudtrail.user_identity.access_key_id`*::
++
+--
+The access key ID that was used to sign the request.
+
+type: keyword
+
+--
+
+[float]
+=== session_context
+
+If the request was made with temporary security credentials, an element that provides information about the session that was created for those credentials
+
+
+*`aws.cloudtrail.user_identity.session_context.mfa_authenticated`*::
++
+--
+The value is true if the root user or IAM user whose credentials were used for the request also was authenticated with an MFA device; otherwise, false.
+
+type: keyword
+
+--
+
+*`aws.cloudtrail.user_identity.session_context.creation_date`*::
++
+--
+The date and time when the temporary security credentials were issued.
+
+type: date
+
+--
+
+*`aws.cloudtrail.user_identity.invoked_by`*::
++
+--
+The name of the AWS service that made the request, such as Amazon EC2 Auto Scaling or AWS Elastic Beanstalk.
+
+type: keyword
+
+--
+
+*`aws.cloudtrail.error_code`*::
++
+--
+The AWS service error if the request returns an error.
+
+type: keyword
+
+--
+
+*`aws.cloudtrail.error_message`*::
++
+--
+If the request returns an error, the description of the error.
+
+type: keyword
+
+--
+
+*`aws.cloudtrail.request_parameters`*::
++
+--
+The parameters, if any, that were sent with the request.
+
+type: keyword
+
+--
+
+*`aws.cloudtrail.response_elements`*::
++
+--
+The response element for actions that make changes (create, update, or delete actions).
+
+type: keyword
+
+--
+
+*`aws.cloudtrail.additional_eventdata`*::
++
+--
+Additional data about the event that was not part of the request or response.
+
+type: keyword
+
+--
+
+*`aws.cloudtrail.request_id`*::
++
+--
+The value that identifies the request. The service being called generates this value.
+
+type: keyword
+
+--
+
+*`aws.cloudtrail.event_type`*::
++
+--
+Identifies the type of event that generated the event record.
+
+type: keyword
+
+--
+
+*`aws.cloudtrail.api_version`*::
++
+--
+Identifies the API version associated with the AwsApiCall eventType value.
+
+type: keyword
+
+--
+
+*`aws.cloudtrail.management_event`*::
++
+--
+A Boolean value that identifies whether the event is a management event.
+
+type: keyword
+
+--
+
+*`aws.cloudtrail.read_only`*::
++
+--
+Identifies whether this operation is a read-only operation.
+
+type: keyword
+
+--
+
+[float]
+=== resources
+
+A list of resources accessed in the event.
+
+
+*`aws.cloudtrail.resources.arn`*::
++
+--
+Resource ARNs
+
+type: keyword
+
+--
+
+*`aws.cloudtrail.resources.account_id`*::
++
+--
+Account ID of the resource owner
+
+type: keyword
+
+--
+
+*`aws.cloudtrail.resources.type`*::
++
+--
+Resource type identifier in the format: AWS::aws-service-name::data-type-name
+
+type: keyword
+
+--
+
+*`aws.cloudtrail.recipient_account_id`*::
++
+--
+Represents the account ID that received this event.
+
+type: keyword
+
+--
+
+*`aws.cloudtrail.service_event_details`*::
++
+--
+Identifies the service event, including what triggered the event and the result.
+
+type: keyword
+
+--
+
+*`aws.cloudtrail.shared_event_id`*::
++
+--
+GUID generated by CloudTrail to uniquely identify CloudTrail events from the same AWS action that is sent to different AWS accounts.
+
+type: keyword
+
+--
+
+*`aws.cloudtrail.vpc_endpoint_id`*::
++
+--
+Identifies the VPC endpoint in which requests were made from a VPC to another AWS service, such as Amazon S3.
+
+type: keyword
+
+--
+
 [float]
 === elb
 

filebeat/docs/modules/aws.asciidoc

@@ -14,12 +14,13 @@ beta[]
 
 This is a module for aws logs. It uses filebeat s3 input to get log files from
 AWS S3 buckets with SQS notification. This module supports reading s3 server
-access logs with `s3access` fileset, ELB access logs with `elb` fileset and VPC
-flow logs with `vpc` fileset.
+access logs with `s3access` fileset, ELB access logs with `elb` fileset, VPC
+flow logs with `vpc` fileset, and CloudTrail logs with `cloudtrail` fileset.
 
 Access logs contain detailed information about the requests made to these
 services. VPC flow logs captures information about the IP traffic going to and
-from network interfaces in AWS VPC.
+from network interfaces in AWS VPC.  CloudTrail logs contain events
+that represent actions taken by a user, role or AWS service.
 
 [float]
 === Example dashboard
@@ -62,6 +63,15 @@ Example config:
     # AWS SQS queue url
     #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
 
+    # Profile name for aws credential
+      #var.credential_profile_name: fb-aws
+
+  cloudtrail:
+    enabled: false
+
+    # AWS SQS queue url
+    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
+
     # Profile name for aws credential
     #var.credential_profile_name: fb-aws
 ----
@@ -74,6 +84,12 @@ AWS SQS queue url.
 
 AWS credential profile name.
 
+=== CloudTrail fileset
+
+The `cloudtrail` fileset does not read the CloudTrail Digest files
+that are delivered to the S3 bucket when Log File Integrity is turned
+on, it only reads the CloudTrail logs.
+
 
 [float]
 === Fields

x-pack/filebeat/filebeat.reference.yml

@@ -123,6 +123,15 @@ filebeat.modules:
     # Profile name for aws credential
     #var.credential_profile_name: fb-aws
 
+  cloudtrail:
+    enabled: false
+
+    # AWS SQS queue url
+    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
+
+    # Profile name for aws credential
+    #var.credential_profile_name: fb-aws
+
 #-------------------------------- Azure Module --------------------------------
 - module: azure
   # All logs

x-pack/filebeat/module/aws/_meta/config.yml

@@ -25,3 +25,12 @@
 
     # Profile name for aws credential
     #var.credential_profile_name: fb-aws
+
+  cloudtrail:
+    enabled: false
+
+    # AWS SQS queue url
+    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
+
+    # Profile name for aws credential
+    #var.credential_profile_name: fb-aws

x-pack/filebeat/module/aws/_meta/docs.asciidoc

@@ -9,12 +9,13 @@ beta[]
 
 This is a module for aws logs. It uses filebeat s3 input to get log files from
 AWS S3 buckets with SQS notification. This module supports reading s3 server
-access logs with `s3access` fileset, ELB access logs with `elb` fileset and VPC
-flow logs with `vpc` fileset.
+access logs with `s3access` fileset, ELB access logs with `elb` fileset, VPC
+flow logs with `vpc` fileset, and CloudTrail logs with `cloudtrail` fileset.
 
 Access logs contain detailed information about the requests made to these
 services. VPC flow logs captures information about the IP traffic going to and
-from network interfaces in AWS VPC.
+from network interfaces in AWS VPC.  CloudTrail logs contain events
+that represent actions taken by a user, role or AWS service.
 
 [float]
 === Example dashboard
@@ -57,6 +58,15 @@ Example config:
     # AWS SQS queue url
     #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
 
+    # Profile name for aws credential
+      #var.credential_profile_name: fb-aws
+
+  cloudtrail:
+    enabled: false
+
+    # AWS SQS queue url
+    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
+
     # Profile name for aws credential
     #var.credential_profile_name: fb-aws
 ----
@@ -68,3 +78,9 @@ AWS SQS queue url.
 *`var.credential_profile_name`*::
 
 AWS credential profile name.
+
+=== CloudTrail fileset
+
+The `cloudtrail` fileset does not read the CloudTrail Digest files
+that are delivered to the S3 bucket when Log File Integrity is turned
+on, it only reads the CloudTrail logs.