Add S3 input to retrieve logs from AWS S3 buckets

[kaiyan-sheng]

Different logs from different services can be stored in S3. For example:

• S3 server access logs: provides detailed records for the requests that are made to a bucket.
• VPC flow logs: records for all of the monitored network interfaces are published to a series of log file objects that are stored in the bucket.
• ELB access logs: capture detailed information about requests sent to the load balancer. Each log contains information such as the time the request was received, the client's IP address, latencies, request paths, and server responses.
• Cloudwatch logs: users can choose to export all data from an amazon cloudwatch log group to a specific s3 bucket.

With all the different logs in S3 from different services, it will be good to have a dedicated Filebeat input to retrieve raw lines from S3 objects. To avoid significant lagging with a polling-based s3 only input, we agree on a combination of notification-based and polling-based approach: s3-sqs filebeat input. This requires extra setups in AWS S3 and SQS to add a notification configuration requesting S3 to publish specific type of events to SQS queue.

Right now with this PR, when s3 input is enabled, you can start seeing events in ES for log messages that are retrieved from S3 buckets:

{
  "_index": "filebeat-8.0.0-2019.07.12-000001",
  "_type": "_doc",
  "_id": "efb4287666",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-07-12T19:50:30.106Z",
    "aws": {
      "s3": {
        "bucket": {
          "name": "test-s3-ks-2",
          "arn": "arn:aws:s3:::test-s3-ks-2"
        },
        "object.key": "test-log-12.txt"
      }
    },
    "cloud": {
      "region": "ap-southeast-1",
      "provider": "aws"
    },
    "input": {
      "type": "s3"
    },
    "ecs": {
      "version": "1.0.1"
    },
    "host": {
      "architecture": "x86_64",
      "name": "KaiyanMacBookPro",
      "os": {
        "kernel": "17.7.0",
        "build": "17G7024",
        "platform": "darwin",
        "version": "10.13.6",
        "family": "darwin",
        "name": "Mac OS X"
      },
      "id": "9C7FAB7B-29D1-5926-8E84-158A9CA3E25D",
      "hostname": "KaiyanMacBookPro"
    },
    "agent": {
      "hostname": "KaiyanMacBookPro",
      "id": "7578d49c-6588-4843-85cc-ad3859f99ed1",
      "version": "8.0.0",
      "type": "filebeat",
      "ephemeral_id": "6b32568f-d728-49b4-b5bb-db0157fd3102"
    },
    "message": "test12\n",
    "log": {
      "offset": 7,
      "file.path": "https://test-s3-ks-2.s3-ap-southeast-1.amazonaws.com/test-log-12.txt"
    }
  },
  "fields": {
    "@timestamp": [
      "2019-07-12T19:50:30.106Z"
    ],
    "suricata.eve.timestamp": [
      "2019-07-12T19:50:30.106Z"
    ]
  },
  "sort": [
    1562961030106
  ]
}

For configuration:

filebeat.inputs:
- type: s3
  queue_url: https://sqs.ap-southeast-1.amazonaws.com/627959692251/test-s3-notification

Ideally with this config, s3 input will go to the specified queueURLs to get messages and read them. If the message has eventSource == aws:s3 && eventName == ObjectCreated:Put and bucket.name is included by bucketNames from the config, then read the S3 object that's specified in this SQS message.

[exekias]

Nice to see this going on! I left a few comments & questions

[kaiyan-sheng]

Question: When sqs message points to a s3 object that is not readable or does not exist anymore, we get error message s3/input.go:220 s3 get object request failed: NoSuchKey: The specified key does not exist..
Do we still report an event with this error message or skip it when error occurs?

[exekias]

Question: When sqs message points to a s3 object that is not readable or does not exist anymore, we get error message s3/input.go:220 s3 get object request failed: NoSuchKey: The specified key does not exist..
Do we still report an event with this error message or skip it when error occurs?

This sounds like the kind of error I would like to be notified about, probably something is not configured well if this happens?

[kaiyan-sheng]

Question: When sqs message points to a s3 object that is not readable or does not exist anymore, we get error message s3/input.go:220 s3 get object request failed: NoSuchKey: The specified key does not exist..
Do we still report an event with this error message or skip it when error occurs?

This sounds like the kind of error I would like to be notified about, probably something is not configured well if this happens?

I saw it happen when:

1. I didn't delete the old sqs messages but did delete the old objects from S3. So sqs message is pointing the s3 filebeat input to go read an object/file that doesn't exist anymore. This should not happen(hopefully) in reality because I was just doing cleaning manually.
2. This also happened when I manually uploaded a .png file to s3 bucket. Then sqs got an message saying there is a new object created in s3. But when filebeat goes to read that object, it's not readable because it's a .png.

Maybe I should have a WARN log message when this happens and report an event with empty message field?

[exekias]

It sounds like this should be at ERROR level, as it's really something the user should look into. As we don't have any message retrieved I don't think we need to send anything to the output.

[kaiyan-sheng]

It sounds like this should be at ERROR level, as it's really something the user should look into. As we don't have any message retrieved I don't think we need to send anything to the output.

Sounds good to me! Thanks! This is what I currently have so I will keep the same behavior :-)

[Conky5]

jenkins test this please

[kaiyan-sheng]

Jenkins, test this please

[andrewkroh]

I'm excited to see this feature. I left a few minor issues and some questions.