[Filebeat] Filebeat AWS module cloudtrail fileset not skip the digest log

[sophiaxu8]

• Version: 7.9.0
• Operating System:
• Discuss Forum URL:
• Steps to Reproduce:

Currently the path of cloudtrail is as follows:
source_vpc -> s3 -> SNS -> SQS -> filebeat -> elasticsearch

There are some errors related to the cloudtrail digest log in filebeat log:

2020-08-28T22:49:05.848Z ERROR [s3] s3/input.go:562 key 'Records' not found

2020-08-28T22:49:05.848Z ERROR [s3] s3/input.go:464 decodeJSONWithKey failed for 'AWSLogs/<acct-id>/CloudTrail-Digest/eu-west-3/2020/08/28/835404217325_CloudTrail-Digest_eu-west-3_XXXX_Infosec_Audit_us-west-2_20200828T220732Z.json.gz' from S3 bucket 'XXXX-cloudtrail-logs': key 'Records' not found

2020-08-28T22:49:05.848Z ERROR [s3] s3/input.go:396 createEventsFromS3Info failed processing file from s3 bucket "XXXX-cloudtrail-logs" with name "AWSLogs/<acct-id>/CloudTrail-Digest/eu-west-3/2020/08/28/835404217325_CloudTrail-Digest_eu-west-3_XXXX_Infosec_Audit_us-west-2_20200828T220732Z.json.gz": decodeJSONWithKey failed for 'AWSLogs/<acct-id>/CloudTrail-Digest/eu-west-3/2020/08/28/835404217325_CloudTrail-Digest_eu-west-3_XXXX_Infosec_Audit_us-west-2_20200828T220732Z.json.gz' from S3 bucket 'XXXX-cloudtrail-logs': key 'Records' not found

As mentioned in the document for filebeat AWS module cloudtrail fileset :

The cloudtrail fileset does not read the CloudTrail Digest files that are delivered to the S3 bucket when Log File Integrity is turned on, it only reads the CloudTrail logs.

But filebeat log shows it try to parse the digest log and failed, the expect behaviour is to skip the digest log.

Here is the filebeat.yml config:

setup.dashboards.enabled: false
setup.template.enabled: false
setup.ilm.enabled: false
setup.template.name: ${INDEX}
setup.template.pattern: ${INDEX}-*

monitoring:
  enabled: yes
  elasticsearch:
    hosts:
      - ${ES_SERVER}
    password: ${BEATS_PASSWORD}
    username: ${BEATS_USERNAME}

filebeat.modules:
  - module: aws
    elb:
      enabled: false
    s3access:
      enabled: false
    cloudwatch:
      enabled: false
    ec2:
      enabled: false
    vpcflow:
      enabled: false
    cloudtrail:
      enabled: true
      var.queue_url: ${QUEUE_URL:}
      var.role_arn: ${ROLE_ARN:}
      var.visibility_timeout: ${VISIBILITY_TIMEOUT:300s}

processors:
  - add_fields:
      target: ''
      fields:
        labels.owners:
          - marty.chong
          - platsec
        labels.description: Cloudtrail logs

logging.level: ${LOGGING_LEVEL:info}

output.elasticsearch:
  index: ${INDEX}
  hosts: '${ES_SERVER}'
  username: '${ES_USERNAME}'
  password: '${ES_PASSWORD}'
  ssl.enabled: ${SSL_ENABLED:true}
  ssl.verification_mode: ${SSL_VERIFICATION_MODE:full}
~                                                      

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[leehinman]

Adding Prefix AWSLogs/$AWS_ACCOUNT_ID/CloudTrail/ to the S3 notification will make it so only notifications for the CloudTrail logs are sent to the SQS queue and processed by Filebeat.

For some this may not work because there are too many AWS_ACCOUNT_IDs that are writing to the same S3 bucket.

[leehinman]

closed by #21086