k8s labels should be plain text, but parsed as nested objects?

[antonyc]

vector 0.16.1

Vector Configuration File

  transforms:
  ... not modifying .kubernetes ...
  sources:
    controlplane_logs:
      type: kubernetes_logs
  sinks:
    elastic:
      type: elasticsearch  # https://vector.dev/docs/reference/configuration/sinks/elasticsearch/
      inputs:
        - log_presink_hook
      ....

Debug Output

Expected Behavior

Given 2 pods with label:

• pod1: app=prometheus
• pod2: app.kubernetes.io/instance=prometheus-for-amp

We send to elasticsearch in kubernetes.pod_labels.app:

• pod1: : {"kubernetes": {"pod_labels": {"app": "prometheus"}}}
• pod2: {"kubernetes": {"pod_labels": {"app.kubernetes.io/instance": "prometheus-for-amp"}}}

Actual Behavior

It seems that we get:
We send to elasticsearch in kubernetes.pod_labels.app:

• pod1: : {"kubernetes": {"pod_labels": {"app": "prometheus"}}}
• pod2: {"kubernetes": {"pod_labels": {"app": {"kubernetes": {"io/instance": "prometheus-for-amp"}}}}}

Additional Context

This results in elastic search:
for pod1 index schema is:

            "pod_labels" : {
              "properties" : {
                "app" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },

for pod2:

"pod_labels" : {
              "properties" : {
                "app" : {
                  "properties" : {
                    "kubernetes" : {
                      "properties" : {
                        "io/component" : {
                          "type" : "text",
                          "fields" : {
                            "keyword" : {
                              "type" : "keyword",
                              "ignore_above" : 256
                            }
                          }
                        },
                        "io/instance" : {
                          "type" : "text",
                          "fields" : {
                            "keyword" : {
                              "type" : "keyword",
                              "ignore_above" : 256
                            }
                          }
                        },
                        "io/managed-by" : {
                          "type" : "text",
                          "fields" : {
                            "keyword" : {
                              "type" : "keyword",
                              "ignore_above" : 256
                            }
                          }
                        },
                        "io/name" : {
                          "type" : "text",
                          "fields" : {
                            "keyword" : {
                              "type" : "keyword",
                              "ignore_above" : 256
                            }
                          }
                        },
                        "io/part-of" : {
                          "type" : "text",
                          "fields" : {
                            "keyword" : {
                              "type" : "keyword",
                              "ignore_above" : 256
                            }
                          }
                        },
                        "io/version" : {
                          "type" : "text",
                          "fields" : {
                            "keyword" : {
                              "type" : "keyword",
                              "ignore_above" : 256
                            }
                          }
                        }
                      }
                    }
                  }
                },
                "helm" : {
                  "properties" : {
                    "sh/chart" : {
                      "type" : "text",
                      "fields" : {
                        "keyword" : {
                          "type" : "keyword",
                          "ignore_above" : 256
                        }
                      }
                    }
                  }
                },

[spencergilbert]

I believe this is the same issue as seen here: #8058

ES using dynamic mapping will read fields with "dots" as nested objects, causing the mapping conflicts. As far as I know this is on the ES side, and we're not able to build a payload with dotted fields that wouldn't end up nested.

[antonyc]

I fixed this on vector side with:

tests:
  - name: kub app label moved if "app.kubernetes.io/name" not exists
    inputs:
    - type: log
      insert_at: log_presink_hook
      log_fields:
        kubernetes.pod_labels.app: "test-name"
    outputs:
      - extract_from: log_presink_hook
        conditions:
          - exists(.kubernetes.pod_labels.app) == false
          - .kubernetes.pod_labels."app.kubernetes.io/name" == "test-name"
  - name: kub app label erased if "app.kubernetes.io/name" exists
    inputs:
    - type: log
      insert_at: log_presink_hook
      log_fields:
        kubernetes.pod_labels.app: "test-name"
        kubernetes.pod_labels.app\.kubernetes\.io/name: "set"
    outputs:
      - extract_from: log_presink_hook
        conditions:
          - exists(.kubernetes.pod_labels.app) == false
          - .kubernetes.pod_labels."app.kubernetes.io/name" == "set"

    log_presink_hook:
      type: remap
      inputs: []
      source: |
        if is_string(.kubernetes.pod_labels.app) {
          if !exists(.kubernetes.pod_labels."app.kubernetes.io/name") {
            .kubernetes.pod_labels."app.kubernetes.io/name" = .kubernetes.pod_labels.app
          }
          del(.kubernetes.pod_labels.app)
        }

[spencergilbert]

That'd certainly be one way to do it - assuming app is the only "top level" key causing issues 👍

If there are more I'd suggest replacing the . with _ via lua today, and by VRL when iteration support lands.