Memory leak with simple instance prometheus-0.17.0.linux-amd64

[hvnsweeting]

I'm running a single instance of prometheus-0.17.0.linux-amd64, with below config:

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    target_groups:
      - targets: ['localhost:9090']
  - job_name: 'prometheus_system'
    scrape_interval: 5s
    target_groups:
      - targets: ['localhost:9100']
rule_files:
  - 'alert.rules'

rules:

# cat alert.rules
ALERT memfree_alert_hvn
IF node_memory_MemFree > 10000000

ALERT free_disk
IF (node_filesystem_avail{mountpoint="/", device="/dev/disk/by-label/DOROOT"} / node_filesystem_size * 100) < 20

I ran it with this command:

./prometheus -config.file=config.yml -alertmanager.url=http://localhost:9093

This is graph that show the process leaked memory

[juliusv]

That's likely not "leaked" memory, but Prometheus's internal chunk buffers slowly filling up (it should level out at some point, but your graph is still fairly low, at 400MB).

If you don't have enough RAM, try tuning the flags mentioned in http://prometheus.io/docs/operating/storage/#memory-usage. Especially storage.local.memory-chunks.

[juliusv]

You can also graph the number of current memory chunks by graphing prometheus_local_storage_memory_chunks, so you can compare that to your memory graph. The default value for the maximum number of memory chunks is 1048576, which will result in a RAM usage of around 3GB.

[hvnsweeting]

Thank you for your explanation, I've updated the starting doc to help next user not get in the same
problem.
prometheus/docs#380

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.