Add support for nested partition_key

[falmotlag]

Add support for nested key to be used as partition_key. This is needed in cases where the target partition_key is nested under another key.

For example, when using this plugin in combination with fluent-bit Kubernetes filter, the filter enriches the log with Kubernetes related keys/values. These enriched entires are added under one key: kubernetes. Heres a log example:

{
    "kubernetes": {
        "annotations": {
            "kubernetes.io/psp": "eks.privileged"
        },
        "container_hash": "kong@sha256:HASH",
        "container_image": "kong:2.0",
        "container_name": "proxy",
        "docker_id": "4bbce70674ca970382d4440a125fbbfcafd123bae5fd50a41e239e3867c5b645",
        "host": "ip.compute.internal",
        "labels": {
            "app.kubernetes.io/component": "app",
            "app.kubernetes.io/instance": "kong"
        },
        "namespace_name": "default",
        "pod_id": "c331cafa-407e-4d7e-8a89-7fc3e1752825",
        "pod_name": "kong-kong-8675766d4f-bbt9b"
    },
    "log": "192.168.112.243 - - [13/Jun/2020:13:23:00 +0000] \"POST /spywall/timeConfig.php HTTP/1.1\" 400 12 \"-\" \"XTC\"\n",
    "stream": "stdout",
    "time": "2020-06-13T13:23:00.637148845Z"
}

Consider the case where docker_id is used as a partition_key. Currently, this plugin (version: 1.2.2) does not support such case. kubernetes.docker_id cannot be used.

[jtackaberry]

I suppose I'm the first person on the planet to use the AWS for Fluent Bit Helm chart with Kinesis data streams. :)

The helm chart in its current state combined with this issue actually makes it undeployable. I've submitted a merge request for the chart to allow partition_key to be optional, so that it defaults to a dynamically created partition key and therefore is at least usable.

But this really does need to be implemented because at the moment with the K8s filter there are no top-level keys suitable to be used for partition routing.

[falmotlag]

@jtackaberry which helm chart are you referring to?

[jtackaberry]

@jtackaberry which helm chart are you referring to?

https://github.com/aws/eks-charts/tree/master/stable/aws-for-fluent-bit - the pull request in question is referenced above my last post.

[lubingfeng]

This is offered as part of https://github.com/aws/aws-for-fluent-bit/releases/tag/v2.9.0.