prometheus data generate too quickly

[FLYyyyy2016]

Bug Report

I found out that my prometheus-docker error has occurred. The error is manifested in prometheus generating a large number of tmp folders. This folder will quickly take up disk space (500M per folder, one per minute), and finally cause the system to crash.

How to solve the problem of quickly generating tmp folder

use ubuntu16.

• System information:
  Linux 4.4.0-116-generic x86_64

• Prometheus version:

prometheus, version 2.0.0 (branch: HEAD, revision: 0a74f98)
build user: root@615b82cb36b6
build date: 20171108-07:11:59
go version: go1.9.2

• Prometheus configuration file:

global:
  evaluation_interval: 15s
  external_labels:
    region: 'region1' 
  scrape_interval: 15s
  scrape_timeout: 15s
alerting:
  alertmanagers:
    # docker_swarm_alertmanager
    - dns_sd_configs:
      - names: [tasks.monitoring_alertmanager]
        type: A
        port: 9093
    - static_configs:
      - targets:
        - 10.0.10.16:9093

rule_files:
- alerts.yml

scrape_configs:
  - job_name: fluentd
    scheme: http
    metrics_path: /metrics
    honor_labels: False
    scrape_interval: 15s
    scrape_timeout: 15s
    static_configs:
    - targets: ['10.0.144.124:24231','10.0.144.41:24231','10.0.144.17:24231','10.0.144.91:24231','10.0.144.73:24231','10.0.144.81:24231','10.0.144.134:24231','10.0.144.11:24231','10.0.144.112:24231','10.0.144.123:24231','10.0.144.105:24231','10.0.144.31:24231','10.0.136.159:24231','10.0.144.32:24231','10.0.144.22:24231','10.0.144.141:24231','10.0.144.131:24231','10.0.144.128:24231','10.0.136.151:24231','10.0.144.135:24231','10.0.144.102:24231','10.0.144.125:24231','10.0.144.51:24231','10.0.144.138:24231','10.0.144.115:24231','10.0.144.42:24231','10.0.144.33:24231','10.0.136.153:24231','10.0.144.18:24231','10.0.144.121:24231','10.0.136.161:24231','10.0.136.154:24231','10.0.144.71:24231','10.0.144.92:24231','10.0.144.140:24231','10.0.144.117:24231','10.0.144.137:24231','10.0.144.118:24231','10.0.144.106:24231','10.0.144.63:24231','10.0.144.83:24231','10.0.136.152:24231','10.0.144.127:24231','10.0.144.82:24231','10.0.144.133:24231','10.0.144.122:24231','10.0.144.55:24231','10.0.144.62:24231','10.0.144.130:24231','10.0.144.16:24231','10.0.144.21:24231','10.0.144.139:24231','10.0.144.116:24231','10.0.144.52:24231','10.0.144.113:24231','10.0.144.111:24231','10.0.144.114:24231','10.0.144.93:24231','10.0.144.43:24231','10.0.144.119:24231','10.0.144.136:24231','10.0.144.72:24231','10.0.144.61:24231','10.0.144.129:24231','10.0.144.57:24231','10.0.144.23:24231','10.0.136.155:24231','10.0.136.156:24231','10.0.136.158:24231','10.0.144.132:24231','10.0.144.101:24231','10.0.136.160:24231','10.0.144.103:24231','10.0.144.126:24231','10.0.136.157:24231']
    relabel_configs:
      - source_labels: ['__address__']
        regex: "10.0.144.124:24231"
        target_label: "host"
        replacement: "cmp004"
      - source_labels: ['__address__']
        regex: "10.0.144.41:24231"
        target_label: "host"
        replacement: "dbs01"
      - source_labels: ['__address__']
        regex: "10.0.144.17:24231"
        target_label: "host"
        replacement: "cid02"
      - source_labels: ['__address__']
        regex: "10.0.144.91:24231"

• Logs:

monitoring_server.0.tcxqhx32fotm@mon01    | level=info ts=2019-02-22T09:47:43.360865888Z caller=compact.go:361 component=tsdb msg="compact blocks" count=1 mint=1550743200000 maxt=1550750400000
monitoring_server.0.tcxqhx32fotm@mon01    | level=error ts=2019-02-22T09:47:44.155533541Z caller=db.go:260 component=tsdb msg="compaction failed" err="persist head block: write compaction: add series: out-of-order series added with label set \"{__name__=\\\"ALERTS\\\",alertname=\\\"PrometheusTargetDown\\\",alertstate=\\\"pending\\\",instance=\\\"10.0.0.11:9090\\\",job=\\\"prometheus\\\",route=\\\"email,salesforce\\\",service=\\\"prometheus\\\",severity=\\\"critical\\\"}\""
monitoring_server.0.tcxqhx32fotm@mon01    | level=info ts=2019-02-22T09:48:44.157157911Z caller=compact.go:361 component=tsdb msg="compact blocks" count=1 mint=1550743200000 maxt=1550750400000
monitoring_server.0.tcxqhx32fotm@mon01    | level=error ts=2019-02-22T09:48:44.920954525Z caller=db.go:260 component=tsdb msg="compaction failed" err="persist head block: write compaction: add series: out-of-order series added with label set \"{__name__=\\\"ALERTS\\\",alertname=\\\"PrometheusTargetDown\\\",alertstate=\\\"pending\\\",instance=\\\"10.0.0.11:9090\\\",job=\\\"prometheus\\\",route=\\\"email,salesforce\\\",service=\\\"prometheus\\\",severity=\\\"critical\\\"}\""
monitoring_server.0.tcxqhx32fotm@mon01    | level=info ts=2019-02-22T09:49:44.922536092Z caller=compact.go:361 component=tsdb msg="compact blocks" count=1 mint=1550743200000 maxt=1550750400000
monitoring_server.0.tcxqhx32fotm@mon01    | level=error ts=2019-02-22T09:49:45.733283161Z caller=db.go:260 component=tsdb msg="compaction failed" err="persist head block: write compaction: add series: out-of-order series added with label set \"{__name__=\\\"ALERTS\\\",alertname=\\\"PrometheusTargetDown\\\",alertstate=\\\"pending\\\",instance=\\\"10.0.0.11:9090\\\",job=\\\"prometheus\\\",route=\\\"email,salesforce\\\",service=\\\"prometheus\\\",severity=\\\"critical\\\"}\""
monitoring_server.0.tcxqhx32fotm@mon01    | level=info ts=2019-02-22T09:50:45.734753683Z caller=compact.go:361 component=tsdb msg="compact blocks" count=1 mint=1550743200000 maxt=1550750400000
monitoring_server.0.tcxqhx32fotm@mon01    | level=error ts=2019-02-22T09:50:46.51924786Z caller=db.go:260 component=tsdb msg="compaction failed" err="persist head block: write compaction: add series: out-of-order series added with label set \"{__name__=\\\"ALERTS\\\",alertname=\\\"PrometheusTargetDown\\\",alertstate=\\\"pending\\\",instance=\\\"10.0.0.11:9090\\\",job=\\\"prometheus\\\",route=\\\"email,salesforce\\\",service=\\\"prometheus\\\",severity=\\\"critical\\\"}\""
monitoring_server.0.tcxqhx32fotm@mon01    | level=info ts=2019-02-22T09:51:46.520808651Z caller=compact.go:361 component=tsdb msg="compact blocks" count=1 mint=1550743200000 maxt=1550750400000
monitoring_server.0.tcxqhx32fotm@mon01    | level=error ts=2019-02-22T09:51:47.277711732Z caller=db.go:260 component=tsdb msg="compaction failed" err="persist head block: write compaction: add series: out-of-order series added with label set \"{__name__=\\\"ALERTS\\\",alertname=\\\"PrometheusTargetDown\\\",alertstate=\\\"pending\\\",instance=\\\"10.0.0.11:9090\\\",job=\\\"prometheus\\\",route=\\\"email,salesforce\\\",service=\\\"prometheus\\\",severity=\\\"critical\\\"}\""
monitoring_server.0.tcxqhx32fotm@mon01    | level=info ts=2019-02-22T09:52:47.279307708Z caller=compact.go:361 component=tsdb msg="compact blocks" count=1 mint=1550743200000 maxt=1550750400000
monitoring_server.0.tcxqhx32fotm@mon01    | level=error ts=2019-02-22T09:52:48.07003226Z caller=db.go:260 component=tsdb msg="compaction failed" err="persist head block: write compaction: add series: out-of-order series added with label set \"{__name__=\\\"ALERTS\\\",alertname=\\\"PrometheusTargetDown\\\",alertstate=\\\"pending\\\",instance=\\\"10.0.0.11:9090\\\",job=\\\"prometheus\\\",route=\\\"email,salesforce\\\",service=\\\"prometheus\\\",severity=\\\"critical\\\"}\""
monitoring_server.0.tcxqhx32fotm@mon01    | level=info ts=2019-02-22T09:53:48.071621088Z caller=compact.go:361 component=tsdb msg="compact blocks" count=1 mint=1550743200000 maxt=1550750400000
monitoring_server.0.tcxqhx32fotm@mon01    | level=error ts=2019-02-22T09:53:48.846985845Z caller=db.go:260 component=tsdb msg="compaction failed" err="persist head block: write compaction: add series: out-of-order series added with label set \"{__name__=\\\"ALERTS\\\",alertname=\\\"PrometheusTargetDown\\\",alertstate=\\\"pending\\\",instance=\\\"10.0.0.11:9090\\\",job=\\\"prometheus\\\",route=\\\"email,salesforce\\\",service=\\\"prometheus\\\",severity=\\\"critical\\\"}\""
monitoring_server.0.tcxqhx32fotm@mon01    | level=info ts=2019-02-22T09:54:48.84874998Z caller=compact.go:361 component=tsdb msg="compact blocks" count=1 mint=1550743200000 maxt=1550750400000
monitoring_server.0.tcxqhx32fotm@mon01    | level=error ts=2019-02-22T09:54:49.610232306Z caller=db.go:260 component=tsdb msg="compaction failed" err="persist head block: write compaction: add series: out-of-order series added with label set \"{__name__=\\\"ALERTS\\\",alertname=\\\"PrometheusTargetDown\\\",alertstate=\\\"pending\\\",instance=\\\"10.0.0.11:9090\\\",job=\\\"prometheus\\\",route=\\\"email,salesforce\\\",service=\\\"prometheus\\\",severity=\\\"critical\\\"}\""
monitoring_server.0.tcxqhx32fotm@mon01    | level=info ts=2019-02-22T09:55:49.611874972Z caller=compact.go:361 component=tsdb msg="compact blocks" count=1 mint=1550743200000 maxt=1550750400000
monitoring_server.0.tcxqhx32fotm@mon01    | level=error ts=2019-02-22T09:55:50.386454828Z caller=db.go:260 component=tsdb msg="compaction failed" err="persist head block: write compaction: add series: out-of-order series added with label set \"{__name__=\\\"ALERTS\\\",alertname=\\\"PrometheusTargetDown\\\",alertstate=\\\"pending\\\",instance=\\\"10.0.0.11:9090\\\",job=\\\"prometheus\\\",route=\\\"email,salesforce\\\",service=\\\"prometheus\\\",severity=\\\"critical\\\"}\""
monitoring_server.0.tcxqhx32fotm@mon01    | 2019-02-22 09:56:19.186 - INFO - Reloading service, new hash e2fbdff7c8747a256355966610f8596db7f6c5b2d94fbc614f37f1a260745bd2
monitoring_server.0.tcxqhx32fotm@mon01    | 2019-02-22 09:56:19.187 - DEBUG - Sending signal 1 to pid 7
monitoring_server.0.tcxqhx32fotm@mon01    | level=info ts=2019-02-22T09:56:19.187520143Z caller=main.go:394 msg="Loading configuration file" filename=/srv/prometheus/prometheus.yml
monitoring_server.0.tcxqhx32fotm@mon01    | 2019-02-22 09:56:39.206 - INFO - Reloading service, new hash 0a051667092f18e1caed62c4087935e38b3d35f1250fb1950377a3baf0e7adf9
monitoring_server.0.tcxqhx32fotm@mon01    | 2019-02-22 09:56:39.206 - DEBUG - Sending signal 1 to pid 7
monitoring_server.0.tcxqhx32fotm@mon01    | level=info ts=2019-02-22T09:56:39.206460667Z caller=main.go:394 msg="Loading configuration file" filename=/srv/prometheus/prometheus.yml
monitoring_server.0.tcxqhx32fotm@mon01    | level=info ts=2019-02-22T09:56:50.388149311Z caller=compact.go:361 component=tsdb msg="compact blocks" count=1 mint=1550743200000 maxt=1550750400000
monitoring_server.0.tcxqhx32fotm@mon01    | level=error ts=2019-02-22T09:56:51.163314718Z caller=db.go:260 component=tsdb msg="compaction failed" err="persist head block: write compaction: add series: out-of-order series added with label set \"{__name__=\\\"ALERTS\\\",alertname=\\\"PrometheusTargetDown\\\",alertstate=\\\"pending\\\",instance=\\\"10.0.0.11:9090\\\",job=\\\"prometheus\\\",route=\\\"email,salesforce\\\",service=\\\"prometheus\\\",severity=\\\"critical\\\"}\""
monitoring_server.0.tcxqhx32fotm@mon01    | level=info ts=2019-02-22T09:57:51.167502189Z caller=compact.go:361 component=tsdb msg="compact blocks" count=1 mint=1550743200000 maxt=1550750400000
monitoring_server.0.tcxqhx32fotm@mon01    | level=error ts=2019-02-22T09:57:51.945684788Z caller=db.go:260 component=tsdb msg="compaction failed" err="persist head block: write compaction: add series: out-of-order series added with label set \"{__name__=\\\"ALERTS\\\",alertname=\\\"PrometheusTargetDown\\\",alertstate=\\\"pending\\\",instance=\\\"10.0.0.11:9090\\\",job=\\\"prometheus\\\",route=\\\"email,salesforce\\\",service=\\\"prometheus\\\",severity=\\\"critical\\\"}\""
monitoring_server.0.tcxqhx32fotm@mon01    | level=info ts=2019-02-22T09:58:51.947502098Z caller=compact.go:361 component=tsdb msg="compact blocks" count=1 mint=1550743200000 maxt=1550750400000
monitoring_server.0.tcxqhx32fotm@mon01    | level=error ts=2019-02-22T09:58:52.720529177Z caller=db.go:260 component=tsdb msg="compaction failed" err="persist head block: write compaction: add series: out-of-order series added with label set \"{__name__=\\\"ALERTS\\\",alertname=\\\"PrometheusTargetDown\\\",alertstate=\\\"pending\\\",instance=\\\"10.0.0.11:9090\\\",job=\\\"prometheus\\\",route=\\\"email,salesforce\\\",service=\\\"prometheus\\\",severity=\\\"critical\\\"}\""

[simonpasquier]

This sounds a lot like what prometheus/tsdb#512 fixes (it should be available in v2.8). In any case, you're running v2.0.0 which is quite old so to rule out other issues, can you update to v2.7.1 at least?