Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] S3 input issue for cloudtrail: createEventsFromS3Info failed #16374

Closed
lliknart opened this issue Feb 17, 2020 · 8 comments · Fixed by #16441 or #17121
Closed

[Filebeat] S3 input issue for cloudtrail: createEventsFromS3Info failed #16374

lliknart opened this issue Feb 17, 2020 · 8 comments · Fixed by #16441 or #17121
Assignees
Labels
bug Team:Platforms Label for the Integrations - Platforms team

Comments

@lliknart
Copy link

I have the following error when aws module is enable

  • createEventsFromS3Info failed
  • gzip.NewReader failed: gzip: invalid header

According to the release notes for beats 7.6.0, it should be fixed

filebeat[8549]: 2020-02-17T16:22:37.354Z        WARN        [s3]        s3/input.go:277        Processing message failed, updating visibility timeout
filebeat[8549]: 2020-02-17T16:22:37.355Z        ERROR        [s3]        s3/input.go:434        gzip.NewReader failed: gzip: invalid header
filebeat[8549]: 2020-02-17T16:22:37.356Z        ERROR        [s3]        s3/input.go:386        createEventsFromS3Info failed for AWSLogs/xxxxxxxxxxxx/CloudTrail/us-east-1/2020/02/15/xxxxxxxxxxxx_CloudTrail_us-east-1_20200215T0900Z_iJSOIOJP6ErPw7vU.json.gz: gzip.NewReader failed: gzip: invalid header
filebeat[8549]: 2020-02-17T16:22:37.356Z        WARN        [s3]        s3/input.go:277        Processing message failed, updating visibility timeout
filebeat[8549]: 2020-02-17T16:22:37.356Z        INFO        [s3]        s3/input.go:282        Message visibility timeout updated to 300
filebeat[8549]: 2020-02-17T16:22:37.357Z        INFO        [s3]        s3/input.go:282        Message visibility timeout updated to 300
filebeat[8549]: 2020-02-17T16:22:37.358Z        INFO        [s3]        s3/input.go:282        Message visibility timeout updated to 300
filebeat[8549]: 2020-02-17T16:22:37.360Z        INFO        [s3]        s3/input.go:282        Message visibility timeout updated to 300
filebeat[8549]: 2020-02-17T16:22:37.365Z        ERROR        [s3]        s3/input.go:434        gzip.NewReader failed: gzip: invalid header
filebeat[8549]: 2020-02-17T16:22:37.365Z        ERROR        [s3]        s3/input.go:386        createEventsFromS3Info failed for AWSLogs/xxxxxxxxxxxx/CloudTrail/eu-west-1/2020/02/15/xxxxxxxxxxxx_CloudTrail_eu-west-1_20200215T1015Z_C9pEYy8aTobIJdDL.json.gz: gzip.NewReader failed: gzip: invalid header
filebeat[8549]: 2020-02-17T16:22:37.365Z        WARN        [s3]        s3/input.go:277        Processing message failed, updating visibility timeout
filebeat[8549]: 2020-02-17T16:22:37.370Z        INFO        [s3]        s3/input.go:282        Message visibility timeout updated to 300
filebeat[8549]: 2020-02-17T16:22:37.390Z        ERROR        [s3]        s3/input.go:434        gzip.NewReader failed: gzip: invalid header
filebeat[8549]: 2020-02-17T16:22:37.390Z        ERROR        [s3]        s3/input.go:386        createEventsFromS3Info failed for AWSLogs/xxxxxxxxxxxx/CloudTrail/ca-central-1/2020/02/15/xxxxxxxxxxxx_CloudTrail_ca-central-1_20200215T1845Z_OY8MOAzSebm6DKum.json.gz: gzip.NewReader failed: gzip: invalid header

For confirmed bugs, please report:

  • Version: 7.6.0
    filebeat version 7.6.0 (amd64), libbeat 7.6.0 [6a23e8f8f30f5001ba344e4e54d8d9cb82cb107c built 2020-02-05 23:06:45 +0000 UTC]
  • Operating System: Ubuntu
  • Steps to Reproduce:
    Hereafter the configuration file used

filebeat.yml

filebeat.config.modules.path: /etc/filebeat/modules.d/*.yml

##==================== Output Logstash settings =====================
output.logstash:
  hosts: ["localhost:5044"]

module aws.yml

  cloudtrail:
    enabled: true

    # AWS SQS queue url
    var.queue_url: https://sqs.region.amazonaws.com/xxxxxxxxxxxx/queue_name
@ycombinator ycombinator added Team:Platforms Label for the Integrations - Platforms team bug labels Feb 18, 2020
@kaiyan-sheng kaiyan-sheng self-assigned this Feb 18, 2020
@kaiyan-sheng
Copy link
Contributor

Thanks for posting this @lliknart ! Could you check what is the Metadata under your CloudTrail logs in S3? For example like this:
Screen Shot 2020-02-19 at 3 02 48 PM

@kaiyan-sheng
Copy link
Contributor

@lliknart Please ignore my previous comment. I was able to reproduce this bug by using the Metadata shown above. I just created a PR to fix this. Thanks!

@nc-andersenb
Copy link

I am also impacted by this bug. Is there any way to implement this as a patch while waiting for the PR to be merged and released? Or must I just wait?

@kaiyan-sheng
Copy link
Contributor

kaiyan-sheng commented Feb 27, 2020

Sorry this PR is merged several hours too late for 7.6.1...
@nc-andersenb Could you build Filebeat locally and use it till next release?

@kaiyan-sheng
Copy link
Contributor

Still failing, reopen the issue.

@nithyagomathi
Copy link

Hi,

I'm having the same issue for Cloudtrail logs from the S3 bucket. But the metadata of cloud trail looks just like you mentioned in the above screenshot. And I'm using Filebeat version 7.6.1

Below is the error:

2020-04-01T17:42:02.454+0530 ERROR [s3] s3/input.go:447 gzip.NewReader failed: gzip: invalid header
2020-04-01T17:42:02.454+0530 ERROR [s3] s3/input.go:386 createEventsFromS3Info failed

@kaiyan-sheng
Copy link
Contributor

@nithyagomathi Hi! Are you using Cloudtrail fileset or s3 input separately? If you can post your config here, that would be great! Or if you prefer in the discuss forum(https://discuss.elastic.co/c/beats/filebeat), that would be even better 😄 Thanks!!

@nithyagomathi
Copy link

nithyagomathi commented Apr 2, 2020

Hi @kaiyan-sheng

This is my config:

> filebeat.inputs:
> - type: s3
>   queue_url: https://sqs.us-west-2.amazonaws.com/XXXXXXXXXX/sqs-name
>   visibility_timeout: 300s
>   credential_profile_name: default
> cloud.id: "cloudid"
> cloud.auth: "elastic:{password}"

And I also created a topic in the forum ([https://discuss.elastic.co/t/filebeat-reading-logs-from-s3/226065])

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Team:Platforms Label for the Integrations - Platforms team
Projects
None yet
6 participants