[Filebeat] Add basic json log format support for s3 input

[kaiyan-sheng]

Currently s3 input in Filebeat only supports txt file format and gzip format. But logs like cloudtrail are in json format. For example:

{"Records": [{
    "eventVersion": "1.0",
    "userIdentity": {
        "type": "IAMUser",
        "principalId": "EX_PRINCIPAL_ID",
        "arn": "arn:aws:iam::123456789012:user/Alice",
        "accessKeyId": "EXAMPLE_KEY_ID",
        "accountId": "123456789012",
        "userName": "Alice"
    },
    "eventTime": "2014-03-06T21:22:54Z",
    "eventSource": "ec2.amazonaws.com",
    "eventName": "StartInstances",
    "awsRegion": "us-east-2",
    "sourceIPAddress": "205.251.233.176",
    "userAgent": "ec2-api-tools 1.6.12.2",
    "requestParameters": {"instancesSet": {"items": [{"instanceId": "i-ebeaf9e2"}]}},
    "responseElements": {"instancesSet": {"items": [{
        "instanceId": "i-ebeaf9e2",
        "currentState": {
            "code": 0,
            "name": "pending"
        },
        "previousState": {
            "code": 80,
            "name": "stopped"
        }
    }]}}
}]}

Please see more json log examples in https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-log-file-examples.html

[leehinman]

Another good example below, it has 2 Cloudtrail event Records

{
    "Records": [
        {
            "eventVersion": "1.07",
            "eventTime": "2019-11-14T00:51:00Z",
            "awsRegion": "us-east-1",
            "eventID": "EXAMPLE8-9621-4d00-b913-beca2EXAMPLE",
            "eventType": "AwsCloudTrailInsight",
            "recipientAccountId": "123456789012",
            "sharedEventID": "EXAMPLE2-1729-42f1-b735-5d8c0EXAMPLE",
            "insightDetails": {
                "state": "Start",
                "eventSource": "ssm.amazonaws.com",
                "eventName": "UpdateInstanceInformation",
                "insightType": "ApiCallRateInsight",
                "insightContext": {
                    "statistics": {
                        "baseline": {
                            "average": 85.4202380952
                        },
                        "insight": {
                            "average": 664
                        }
                    }
                }
            },
            "eventCategory": "Insight"
        },
        {
            "eventVersion": "1.07",
            "eventTime": "2019-11-14T00:52:00Z",
            "awsRegion": "us-east-1",
            "eventID": "EXAMPLEc-28be-486c-8928-49ce6EXAMPLE",
            "eventType": "AwsCloudTrailInsight",
            "recipientAccountId": "123456789012",
            "sharedEventID": "EXAMPLE2-1729-42f1-b735-5d8c0EXAMPLE",
            "insightDetails": {
                "state": "End",
                "eventSource": "ssm.amazonaws.com",
                "eventName": "UpdateInstanceInformation",
                "insightType": "ApiCallRateInsight",
                "insightContext": {
                    "statistics": {
                        "baseline": {
                            "average": 85.4202380952
                        },
                        "insight": {
                            "average": 664
                        },
                        "insightDuration": 1
                    }
                }
            },
            "eventCategory": "Insight"
        }
    ]
}

[kaiyan-sheng]

We decided to use expand_event_list_from_field parameter to specify the JSON field same as Kafka input: https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-input-kafka.html#_expand_event_list_from_field