Memory leak in Kubernetes discovery

[grobie]

Bug Report

What did you do?

Ran Prometheus v2.2.1 plus latest race patches (v2.2.1...f8dcf9b) for several days.

What did you expect to see?

Stable memory footprint.

What did you see instead? Under which circumstances?

Memory leak, daily crashes.

Environment

• System information:

  Linux 4.4.10+soundcloud x86_64

• Prometheus version:

  custom built from f8dcf9b

• Prometheus configuration file:

  22 jobs (17 kubernetes_sd from a big cluster, 1 ec2_sd, 2 dns_sd, 2 static_config)

Profiles

• 16:36 pprof.prometheus.alloc_objects.alloc_space.inuse_objects.inuse_space.001.pb.gz
• 18:32 pprof.prometheus.alloc_objects.alloc_space.inuse_objects.inuse_space.002.pb.gz
• svg 002.svg.gz

pprof top 10 comparison

Showing nodes accounting for 1743.66MB, 36.60% of 4764.52MB total
Dropped 220 nodes (cum <= 23.82MB)
      flat  flat%   sum%        cum   cum%
  502.03MB 10.54% 10.54%   453.03MB  9.51%  github.com/prometheus/prometheus/vendor/github.com/ugorji/go/codec.(*jsonDecDriver).DecodeString
  333.68MB  7.00% 17.54%    27.52MB  0.58%  github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/chunkenc.(*bstream).writeBits
  327.53MB  6.87% 24.41%    29.50MB  0.62%  github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*memSeries).cut
 -294.53MB  6.18% 18.23%  -294.53MB  6.18%  github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/chunkenc.NewXORChunk (inline)
  262.15MB  5.50% 23.74%   269.65MB  5.66%  github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.codecSelfer1234.decResourceList
 -216.12MB  4.54% 19.20%  -216.12MB  4.54%  github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/chunkenc.(*bstream).writeByte (inline)
 -147.56MB  3.10% 16.10%  -147.56MB  3.10%  github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/chunkenc.(*bstream).writeBit (inline)
  134.66MB  2.83% 18.93%   134.66MB  2.83%  reflect.unsafe_NewArray
  132.13MB  2.77% 21.70%   776.30MB 16.29%  github.com/prometheus/prometheus/vendor/k8s.io/client-go/pkg/api/v1.codecSelfer1234.decSliceContainer
  130.57MB  2.74% 24.44%   144.37MB  3.03%  github.com/prometheus/prometheus/scrape.(*scrapeLoop).append

[beorn7]

Built from 2cbba4e now (finally a version without known data races so that we can rule out races as a chaos source). Still seing the memory leak. New interesting finding: The K8s pod targets were not updated (Prometheus tried to scrape pods that didn't exist anymore and failed to scrape the new pods). I could fix that by sending SIGHUP which then also brought the escalating memory usage back to normal levels.

This seems to have to do with something in K8s SD getting stuck.

[beorn7]

Fixed by #4117 .

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.