[kubernetes_logs] Logs are not forwarded if pod has "app" label configured

[Nicarim]

I'm getting an error when my pod has label called "app". I've found about it when I tried to run https://github.com/opsgenie/kubernetes-event-exporter/blob/master/deploy/02-deployment.yaml this deployment. I think it should be pretty easy to replicate given that.

I've fixed it by changing the app label to app_name label.

Vector Version

vector 0.14.0 (x86_64-unknown-linux-gnu 5f3a319 2021-06-03)

Vector Configuration File

I'm using helm setup with following values

sinks:
  elasticsearch:
    type: elasticsearch
    inputs:
      - kubernetes_logs
    index: "index-%F"
    endpoint: "removed"
    auth:
      user: "${ELASTICSEARCH_USERNAME}"
      password: "${ELASTICSEARCH_PASSWORD}"
      strategy: basic

Debug Output

I'm not quite sure how to configure the debug output using helm chart, but can provide if it will be really necessary

Expected Behavior

Logs should be forwarded to elasticsearch

Actual Behavior

Logs were dropping, with error in stdout

│ Jun 28 14:14:35.021 ERROR sink{component_kind="sink" component_name=elasticsearch component_type=elasticsearch}:request{request_id=304}: vector::sinks::util::retries: Not retriable; dropping the request. reaso │
│ n="error type: mapper_parsing_exception, reason: object mapping for [kubernetes.pod_labels.app] tried to parse field [app] as object, but found a concrete value"

Example Data

https://github.com/opsgenie/kubernetes-event-exporter/blob/master/deploy/02-deployment.yaml
Deploy this kubernetes cluster alongside vector, configure elasticsearch sink and the logs are not there.

[spencergilbert]

Hey! This is actually because of how fields with dots (.) are handled, see #3588 for details.

I've previously used a lua transform like this to resolve:

[transforms.dedot_labels]
	type = "lua"
	inputs = ["input"]
	version = "2"

	hooks.process = "process"
	source = """
    function process(event, emit)
        if event.log.kubernetes == nil then
            return
        end
        dedot(event.log.kubernetes.pod_labels)
        emit(event)
    end

    function dedot(map)
        if map == nil then
            return
        end
        local new_map = {}
        local changed_keys = {}
        for k, v in pairs(map) do
            local dedotted = string.gsub(k, "%.", "_")
            if dedotted ~= k then
                new_map[dedotted] = v
                changed_keys[k] = true
            end
        end
        for k in pairs(changed_keys) do
            map[k] = nil
        end
        for k, v in pairs(new_map) do
            map[k] = v
        end
    end
    """

[Nicarim]

The error is hugely misleading then - because apparently after I changed from using app it fixed, even though I changed nothing else and no dots were there. Which . exactly causes issue and why changing the name from app to app_name fixed it?

[spencergilbert]

The error is hugely misleading then - because apparently after I changed from using app it fixed, even though I changed nothing else and no dots were there. Which . exactly causes issue and why changing the name from app to app_name fixed it?

It's definitely tricky on the ES side - when you index a document with a label app.kubernetes.io/name: my-app, ES will interpret that as an object:

app: {
  kubernetes: {
    io/name: "my-app",
  }
}

If this event is received by ES before just app the dynamic mapping is created with app = object, and app: name is "concrete" value, which is rejected by ES.

Does that make more sense?

[Nicarim]

Holy moly
That makes total sense, I really thought the error is originated from vector for some reason. I suppose I will close the task, and leave that issue to index in search engine in case someone has same issue :)
Perhaps there could be some indication in error message that's from ES and not from Vector, but other than that thank you very much for script and blazing fast answer.