Memory leak while using kubernetes_logs source

[karlmartink]

Vector Version

0.11.1-alpine

running on Kubernetes

v1.19.3

Vector Configuration File

    [sources.kube_logs_1]
      type = "kubernetes_logs"
      exclude_paths_glob_patterns = ["**/calico-node/**"]
      annotation_fields.container_name = "container_name"
      annotation_fields.container_image = "container_image"
      annotation_fields.pod_ip = "pod_ip"
      annotation_fields.pod_name = "pod_name"
      annotation_fields.pod_namespace = "namespace_name"
      annotation_fields.pod_node_name = "pod_node_name"
      annotation_fields.pod_uid = "pod_uid"

    [transforms.kube_logs_remapped]
      type = "remap"
      inputs = ["kube_logs_1"]
      source = '''
      .full_message = .message
      .message = "pod_stderr_stdout"
      .source = .pod_node_name
      del(.file)
      del(.pod_node_name)
      del(.kubernetes)
      '''
    [transforms.add_region_fields_to_logs]
      type = "add_fields"
      inputs = ["vector_1", "kube_logs_remapped"]
      fields.env = "<%= @environment %>"
      fields.region = "<%= @region %>"
      fields.region_domain = "<%= @region_domain %>"

    [sinks.graylog_gelf]
      type = "http"
      inputs = ["add_region_fields_to_logs"]
      uri = xxx
      encoding.codec = "ndjson"
      compression = "none"
      batch.max_bytes = 4096
      #batch.timeout_secs = 1
      #buffer.max_events = 1
      #buffer.type = "memory"
      tls.verify_hostname = false

Debug Output

https://gist.github.com/karlmartink/095979bec3dea7d91430c91a842d3927

Expected Behavior

Stable memory usage or slight increase until some maximum level is increased.

Actual Behavior

Containers memory + cpu usage is slowly growing overtime until maximum limits are reached and service is killed while processed events remain the same.

Example Data

POD CPU and Memory usage

Processed events

Additional Context

I am using vector to mostly collect Kubernetes logs and manipulate them with some transforms. After that they get forwarded to graylog via HTTP to GELF input.

[karlmartink]

Meanwhile I updated to vector version 0.12-debian and the memory leak still stayed the same.
I split Vector deployment to two different services:

• vector-kube-collector - Collects logs from Kubernetes and forwards to Vector-aggregator.
• vector-aggregator - Receives logs from kube-collector and forwards to Graylog via. HTTP

The discovery I made is that when I disabled HTTP sink and replaced with Blackhole, the memory leaking stopped.
So it seems the culprit of the leak is HTTP sink.

Running with Blackhole sink

Running with HTTP sink

vector-kube-collector configuration file
`

Read Kubernetes logs from files

[sources.kube_logs_1]
  type = "kubernetes_logs"
  exclude_paths_glob_patterns = ["**/calico-node/**"]
  annotation_fields.container_name = "container_name"
  annotation_fields.container_image = "container_image"
  annotation_fields.pod_ip = "pod_ip"
  annotation_fields.pod_name = "pod_name"
  annotation_fields.pod_namespace = "namespace_name"
  annotation_fields.pod_node_name = "pod_node_name"
  annotation_fields.pod_uid = "pod_uid"

# Transform Kube logs and remove unused fields
[transforms.kube_logs_remapped]
  type = "remap"
  inputs = ["kube_logs_1"]
  source = '''
  .full_message = .message
  .message = "pod_stderr_stdout"
  .source = .pod_node_name
  del(.file)
  del(.pod_node_name)
  del(.kubernetes)
  '''

# Forward logs to vector-aggregator service
[sinks.vector_aggr]
  type = "vector"
  inputs = ["kube_logs_remapped"]
  address = "vector-aggregator.monitoring.svc.cluster.local:31014"`

vector-aggregator configuration file
`
# Receive logs from VECTOR agents
[sources.vector_1]
type = "vector"
address = "${POD_IP}:31014"

# Add environment, region and region_domain fields
[transforms.add_region_fields_to_logs]
  type = "add_fields"
  inputs = ["vector_1"]
  fields.env = "<%= @environment %>"
  fields.region = "<%= @region %>"
  fields.region_domain = "<%= @region_domain %>"

# Forward logs to Graylog
[sinks.blackhole]
  type = "blackhole"
  inputs = ["add_region_fields_to_logs"]

[blt]

I haven't had a chance to triage but this does strike me as relevant to the conversation in #6781.

[karlmartink]

As this memory leak issue is the last blocker for me to deploy vector to my production environment, I continued experimenting with different versions and images. All graphs represent "Average pod memory usage"

0.12-debian + HTTP sink

0.12-debian + Blackhole sink

0.12.1-distroless-libc + HTTP sink

@blt if there is any combination that you would like me to test then let me know.

I will continue experimenting with HTTP sink buffer settings and see if they have any effect on the memory leak.

[karlmartink]

Some further testing revealed, that enabling prometheus metrics with the following configuration made things alot worse... can be related to #6781

[sources.internal_metrics]
      type = "internal_metrics"

[sinks.prom_metrics]
  type = "prometheus_exporter"
  inputs = ["internal_metrics"]
  address = "${POD_IP}:8080"
  default_namespace = "vector"`

0.12.1-distroless-libc + HTTP sink + Prometheus exporter

[KHiis]

+1 with the issue. Is there any expected workaround or anything we could look forward to?

[binarylogic]

@KHiis we are actively investigating and will report back as we learn. We're hoping to get to the root cause next week.

[blt]

@karlmartink @KHiis apologies for the delay in getting back to you and thank you for the detailed issue. Sinking into the blackhole not demonstrating this issue would suggest to me that vector's buffering as it backs up into the HTTP sink, but the growth when you enable prometheus is surprising to me. @karlmartink out of curiosity, if you enable prometheus and sink into blackhole does the issue still persist?

Also, if you have the prometheus sink switched on I would be very curious to know about the following bits of telemetry for each step in your config:

• events_in_total
• events_out_total
• processed_bytes_total

If indeed one of the steps is backing up and causing buffering I'd expect this triplet to fall off steeply for one of them. We are also aware of #6305 where vector will hold file descriptors open if the kubernetes log source is unable to drain quickly enough. Do you have a measure of your open FDs? We have also, in 0.12, introduced debug logging that shows which steps in the config have high load. If you enable debug logging -- -vvv -- can you send me log output with lines that match for utilization? The closer to 1 the component is the busier, so that might give us some clue as well.

[karlmartink]

@blt here is some more telemetry. I grouped the graphs by "component_type" to accurately visualise as I have 3 pods running for vector-aggregator.

0.12.1-distroless-libc + HTTP sink + Prometheus exporter

CPU usage graph for the same period

Some utilization info from debug output:

Mar 22 12:34:05.906 DEBUG transform{component_kind="transform" component_name=add_region_fields_to_logs component_type=add_fields}: vector::utilization: utilization=0.8536889391132572
Mar 22 12:34:05.907 TRACE vector: Beep.
Mar 22 12:34:05.907 DEBUG sink{component_kind="sink" component_name=blackhole component_type=blackhole}: vector::utilization: utilization=0.007456749021618303
Mar 22 12:34:10.907 DEBUG transform{component_kind="transform" component_name=add_region_fields_to_logs component_type=add_fields}: vector::utilization: utilization=0.08562899173450227
Mar 22 12:34:10.907 DEBUG sink{component_kind="sink" component_name=blackhole component_type=blackhole}: vector::utilization: utilization=0.0010800648283539548
Mar 22 12:34:10.907 TRACE vector: Beep.
Mar 22 12:34:20.907 DEBUG transform{component_kind="transform" component_name=add_region_fields_to_logs component_type=add_fields}: vector::utilization: utilization=0.00107175880550649
Mar 22 12:34:20.907 DEBUG sink{component_kind="sink" component_name=blackhole component_type=blackhole}: vector::utilization: utilization=0.0003146771524068778
Mar 22 12:34:25.906 DEBUG transform{component_kind="transform" component_name=add_region_fields_to_logs component_type=add_fields}: vector::utilization: utilization=0.0003256409642432362
Mar 22 12:34:25.908 TRACE vector: Beep.
Mar 22 12:34:25.908 DEBUG sink{component_kind="sink" component_name=blackhole component_type=blackhole}: vector::utilization: utilization=0.00027920071659492116

I enabled prometheus exporter to blackhole sink. Will post results soon.

[karlmartink]

0.12.1-distroless-libc
"internal_metrics" enabled
Kubernetes logs received via. Vector source
Blackhole sink

As it can be seen sinking into blackhole again does not demonstrate the issue.

Unfortunately I don't have open FD metric to provide. But as can be seen on the graphs, kubernetes log collecting works without issues and has zero problems.

[blt]

@karlmartink this is great, thank you. @eeyun it strikes me that this might actually be related to the work you're doing these days.