Make label keys templateable for Loki sink

[ndrpnt]

Current Vector Version

0.12.1

Use-cases

In a standard Kubernetes+Promtail+Loki setup, each Pod's label will result in a Loki label. This doesn't impact cardinality and allow filtering related services.
This behavior is not replicable with Vector because it is currently not possible to set dynamic label keys for the Loki sink.

Attempted Solutions

Include pod labels in the message and parse them at query time within Loki, which is less than ideal

Proposal

Make label keys templateable:

[labels]
a_static_key = "{{ some_field }}" # Before
"{{ some_field }}" = "{{ some_other_field }}" # After

References

Mentionned here #6381 (comment)

[ndrpnt]

Made some progress on this issue, sorry for the delay. I'm afraid the proposed solution isn't enough in the long term. However, it is still a useful workaround as it makes achieving the use case described in the issue possible.

With these changes, it is possible to configure Vector like this:

[sources.kubernetes_logs]
  type = "kubernetes_logs"

[transforms.labels]
  type = "lua"
  inputs = ["kubernetes_logs"]
  version = "2"
  # Transform the log structure so that every pod label is referencable without
  # statically knowing its key. Also make sure the key is a valid Loki label.
  hooks.process = '''
  function (event, emit)
    l = {}
    for k,v in pairs(event.log.kubernetes.pod_labels) do
      l[#l+1] = {key=k:gsub("[^a-zA-Z0-9_:]", "_"), value=v}
    end
    event.log.labels = l
    emit(event)
  end
  '''

[sinks.loki]
  inputs = ["labels"]
  type = "loki"
  endpoint = "http://loki:3100"
  labels.file = "{{ file }}"
  labels.stream = "{{ stream }}"
  labels.source_type = "{{ source_type }}"
  labels.kubernetes_container_id = "{{ kubernetes.container_id }}"
  labels.kubernetes_container_image = "{{ kubernetes.container_image }}"
  labels.kubernetes_container_name = "{{ kubernetes.container_name }}"
  labels.kubernetes_pod_ip = "{{ kubernetes.pod_ip }}"
  labels.kubernetes_pod_name = "{{ kubernetes.pod_name }}"
  labels.kubernetes_pod_namespace = "{{ kubernetes.pod_namespace }}"
  labels.kubernetes_pod_node_name = "{{ kubernetes.pod_node_name }}"
  labels.kubernetes_pod_uid = "{{ kubernetes.pod_uid }}"
  # We take advantage of the fact that Vector drops labels referencing a field
  # that does not exist.
  labels."kubernetes_pod_labels_{{ labels[0].key }}" = "{{ labels[0].value }}"
  labels."kubernetes_pod_labels_{{ labels[1].key }}" = "{{ labels[1].value }}"
  labels."kubernetes_pod_labels_{{ labels[2].key }}" = "{{ labels[2].value }}"
  labels."kubernetes_pod_labels_{{ labels[3].key }}" = "{{ labels[3].value }}"
  labels."kubernetes_pod_labels_{{ labels[4].key }}" = "{{ labels[4].value }}"
  labels."kubernetes_pod_labels_{{ labels[5].key }}" = "{{ labels[5].value }}"
  labels."kubernetes_pod_labels_{{ labels[6].key }}" = "{{ labels[6].value }}"
  labels."kubernetes_pod_labels_{{ labels[7].key }}" = "{{ labels[7].value }}"
  labels."kubernetes_pod_labels_{{ labels[8].key }}" = "{{ labels[8].value }}"
  labels."kubernetes_pod_labels_{{ labels[9].key }}" = "{{ labels[9].value }}"
  remove_label_fields = true
  encoding.codec = "json"

It is ugly in multiple ways but solves my issue when pushing logs with dynamic labels to Loki. I can add some documentation and submit a PR if that's ok for you.

Taking a step back, I think the issue is that Vector in general falls a bit short when it comes to handling dynamic keys, that is, keys that depend on event fields. Keys that are not statically known (or whose count may vary) are currently hard to work with both in config.toml and in VRL.

I hit this limitation a few times with the Loki sink because it is often desirable to to index the logs by some labels that cannot be known when configuring Vector (e.g. Docker image labels or Kubernetes pod labels). Quickly glancing over Vector documentation, it seems that one could stumble upon the same issue when configuring tags in the s3 sink or query parameters in the Elasticsearch sink, among others.

As a side-note, Kubernetes + Loki + Promtail is a setup that seem to gain quite a bit of traction. I'd love to swap Promtail for Vector but this limitation make me think twice. Also, the above transform should ideally be a remap but I don't think it can be done with VRL currently (has to do with dynamic keys here too).

As this limitation affect Vector configuration and VRL in multiple places, in the future, I'd love to have a generic solution to handle dynamic keys. WDYT?

[spencergilbert]

As a side-note, Kubernetes + Loki + Promtail is a setup that seem to gain quite a bit of traction. I'd love to swap Promtail for Vector but this limitation make me think twice. Also, the above transform should ideally be a remap but I don't think it can be done with VRL currently (has to do with dynamic keys here too).

Definitely, in fact... #3588

Definitely agree that having to write out each array item isn't ideal there, @jszwedko do you know if there's been any internal discussion/thoughts around this

[jszwedko]

No discussion that I'm aware of, but I agree that there is a gap here with handling of dynamic keys for templated values like this.