Prometheus panic

[mg03]

What did you do? Run prometheus 1.2.1 as a service

What did you expect to see? Prometheus running

What did you see instead? Under which circumstances? Panic

Environment Ubuntu 14.04

• System information:

  Linux 4.4.0-62-generic x86_64

• Prometheus version:

  1.2.1

• Alertmanager version:

  insert output of alertmanager -version here (if relevant to the issue)

• Prometheus configuration file:

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # By default, scrape targets every 15 seconds.
  # scrape_timeout is set to the global default (10s).

  # The labels to add to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager, etc).
  external_labels:
    apptype: 'svc'
    datacenter: 'us-east-1'

rule_files:
  - /alert_rules/*.rules

# A list of scrape configurations.

scrape_configs:

# gets node_exporter info of all nodes whose services registered in consul
  - job_name: 'node'
    scrape_interval: 10s
    scrape_timeout:  10s
    consul_sd_configs:
      - server: 'localhost:8500'
        datacenter: us-east-1
        services: []
    relabel_configs:
      - source_labels: [__meta_consul_service]
        target_label: stack
      - source_labels: ['__address__']
        regex: '^(.+):.+'
        target_label: __address__
        replacement: '${1}:9100'
      - source_labels: [__meta_consul_address]
        target_label: instance
      - source_labels: [__meta_consul_tags]
        regex: .*,svc,.*
        action: keep
      - source_labels: [__meta_consul_tags]
        regex: .*,prod,.*
        action: keep
      - source_labels: [__meta_consul_tags]
        regex: .*,infra,.*
        action: drop
      - source_labels: [__meta_consul_tags]
        regex: .*,alertmanager,.*
        action: drop
      - source_labels: [__meta_consul_tags]
        regex: .*,pushgateway,.*
        action: drop
      - source_labels: [__meta_consul_tags]
        regex: .*,prometheus,.*
        action: drop
      - source_labels: [__meta_consul_tags]
        regex: .*,blackbox_exporter,.*
        action: drop
      - source_labels: [__meta_consul_tags]
        regex: .*,(prod),.*
        target_label: env
        replacement: '${1}'
      - source_labels: [__meta_consul_tags]
        target_label: plan
        regex: '.*,plan--(.+),.*'
        replacement: '${1}'
      - source_labels: [__meta_consul_service]
        target_label: version
        regex: '^.*--(.+)--.*'
        replacement: '${1}'


  - job_name: 'consul-exporter'
    scrape_interval: 10s
    scrape_timeout:  10s
    static_configs:
      - targets: ['localhost:9150']

#  - job_name: 'prometheus'
#    scrape_interval: 10s
#    scrape_timeout:  10s
#    static_configs:
#      - targets: ['localhost:9090']

  - job_name: 'app_metrics'
    scrape_interval: 10s
    scrape_timeout:  10s
    honor_labels: true
    consul_sd_configs:
      - server: 'localhost:8500'
        datacenter: us-east-1
        services: []
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: .*,svc,.*
        action: keep
      - source_labels: [__meta_consul_tags]
        regex: .*,prod,.*
        action: keep
      - source_labels: [__meta_consul_tags]
        regex: .*,infra,.*
        action: drop
      - source_labels: [__meta_consul_tags]
        regex: .*,alertmanager,.*
        action: drop
      - source_labels: [__meta_consul_tags]
        regex: .*,pushgateway,.*
        action: drop
      - source_labels: [__meta_consul_tags]
        regex: .*,prometheus,.*
        action: drop
      - source_labels: [__meta_consul_tags]
        regex: .*,blackbox_exporter,.*
        action: drop
      - source_labels: [__meta_consul_service]
        target_label: stack
      - source_labels: ['__address__']
        regex: '^(.+):.+'
        target_label: __address__
        replacement: '${1}:5775'
      - source_labels: [__meta_consul_address]
        target_label: instance
      - source_labels: [__meta_consul_tags]
#        regex: .*,(dev|qa|prod),.*
        regex: .*,(prod),.*
        target_label: env
        replacement: '${1}'
      - source_labels: [__meta_consul_service]
        regex: '^.*--(.+)--.*$'
        target_label: version
        replacement: '${1}'
      - source_labels: [__meta_consul_tags]
        regex: .*,plan--(.+),.*
        target_label: plan
        replacement: '${1}'

#  - job_name: 'alertmanager'
#    scrape_interval: 10s
#    scrape_timeout:  10s
#    consul_sd_configs:
#      - server: 'localhost:8500'
#        datacenter: us-east-1
#        services: []
#    relabel_configs:
#      - source_labels: [__meta_consul_service]
#        target_label: stack
#      - source_labels: ['__address__']
#        regex: '^(.+):.+'
#        target_label: __address__
#        replacement: '${1}:9093'
#      - source_labels: [__meta_consul_address]
#        target_label: instance
#      - source_labels: [__meta_consul_tags]
#        regex: .*,alertmanager,.*
#        action: keep
#      - source_labels: [__meta_consul_tags]
#        regex: .*,svc,.*
#        action: keep
#      - source_labels: [__meta_consul_tags]
#        regex: .*,infra,.*
#        action: drop
#      - source_labels: [__meta_consul_tags]
#        regex: .*,(dev|qa|prod),.*
#        target_label: env
#        replacement: '${1}'

  - job_name: pushgateway
    scrape_interval: 10s
    honor_labels: true
    consul_sd_configs:
      - server: 'localhost:8500'
        datacenter: us-east-1
        services: []
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: .*,pushgateway,.*
        action: keep
      - source_labels: [__meta_consul_tags]
        regex: .*,svc,.*
        action: keep
      - source_labels: [__meta_consul_tags]
        regex: .*,infra,.*
        action: drop

  - job_name: 'nomad'
    scrape_interval: 10s
    scrape_timeout:  10s
    consul_sd_configs:
      - server: 'localhost:8500'
        datacenter: us-east-1
        services: ['nomad','nomad-client']
    relabel_configs:
      - source_labels: [__meta_consul_service]
        target_label: stack
      - source_labels: ['__address__']
        regex: '^(.+):.+'
        target_label: __address__
        replacement: '${1}:9102'
      - source_labels: [__meta_consul_address]
        target_label: instance

  - job_name: 'nomad-node'
    scrape_interval: 10s
    scrape_timeout:  10s
    consul_sd_configs:
      - server: 'localhost:8500'
        datacenter: us-east-1
        services: ['nomad','nomad-client']
    relabel_configs:
      - source_labels: [__meta_consul_service]
        target_label: stack
      - source_labels: ['__address__']
        regex: '^(.+):.+'
        target_label: __address__
        replacement: '${1}:9100'
      - source_labels: [__meta_consul_address]
        target_label: instance

  - job_name: 'container'
    scrape_interval: 10s
    scrape_timeout:  10s
    consul_sd_configs:
      - server: 'localhost:8500'
        datacenter: us-east-1
        services: ['nomad','nomad-client']
    relabel_configs:
      - source_labels: ['__address__']
        regex: '^(.+):.+'
        target_label: __address__
        replacement: '${1}:8080'
      - source_labels: [__meta_consul_address]
        target_label: instance

• Alertmanager configuration file:

insert configuration here (if relevant to the issue)

• Logs:

2017-02-24 03:32:47 | prometheus - {"level":"info","msg":"Checkpointing in-memory metrics and chunks...","source":"persistence.go:549","time":"2017-02-24T03:32:47Z"}
2017-02-24 03:32:48 | prometheus - {"chunksToPersist":3136631,"level":"error","maxChunksToPersist":3136569,"maxToleratedMemChunks":10350678,"memoryChunks":9410166,"msg":"Storage needs throttling. Scrapes and rule evaluations will be skipped.","source":"storage.go:888","time":"2017-02-24T03:32:48Z"}
2017-02-24 03:32:49 | prometheus - panic: send on closed channel
2017-02-24 03:32:49 | prometheus -
2017-02-24 03:32:49 | prometheus - goroutine 277839424 [running]:
2017-02-24 03:32:49 | prometheus - panic(0xc4d200, 0xcc73346840)
2017-02-24 03:32:49 | prometheus - /usr/local/go/src/runtime/panic.go:500 +0x1a1
2017-02-24 03:32:49 | prometheus - github.com/prometheus/prometheus/retrieval/discovery/consul.(*consulService).watch(0xc643150bd0, 0x7ff63b3c2270, 0xc758abd2c0, 0xcdfd266060)
2017-02-24 03:32:49 | prometheus - /go/src/github.com/prometheus/prometheus/retrieval/discovery/consul/consul.go:265 +0xb35
2017-02-24 03:32:49 | prometheus - created by github.com/prometheus/prometheus/retrieval/discovery/consul.(*Discovery).Run
2017-02-24 03:32:49 | prometheus - /go/src/github.com/prometheus/prometheus/retrieval/discovery/consul/consul.go:169 +0x821
2017-02-24 03:32:54 | prometheus - {"level":"info","msg":"Starting prometheus (version=1.2.1, branch=master, revision=dd66f2e94b2b662804b9aa1b6a50587b990ba8b7)","source":"main.go:75","time":"2017-02-24T03:32:54Z"}
2017-02-24 03:32:54 | prometheus - {"level":"info","msg":"Build context (go=go1.7.1, user=root@fd9b0daff6bd, date=20161010-15:58:23)","source":"main.go:76","time":"2017-02-24T03:32:54Z"}
2017-02-24 03:32:54 | prometheus - {"level":"info","msg":"Loading configuration file /etc/prometheus/prometheus.yml","source":"main.go:247","time":"2017-02-24T03:32:54Z"}
2017-02-24 03:32:59 | prometheus - {"level":"info","msg":"Loading series map and head chunks...","source":"storage.go:354","time":"2017-02-24T03:32:59Z"}
2017-02-24 03:34:27 | prometheus - {"level":"warning","msg":"Persistence layer appears dirty.","source":"persistence.go:712","time":"2017-02-24T03:34:27Z"}
2017-02-24 03:34:27 | prometheus - {"level":"warning","msg":"Starting crash recovery. Prometheus is inoperational until complete.","source":"crashrecovery.go:40","time":"2017-02-24T03:34:27Z"}
2017-02-24 03:34:27 | prometheus - {"level":"warning","msg":"To avoid crash recovery in the future, shut down Prometheus with SIGTERM or a HTTP POST to /-/quit.","source":"crashrecovery.go:41","time":"2017-02-24T03:34:27Z"}
2017-02-24 03:34:27 | prometheus - {"level":"info","msg":"Scanning files.","source":"crashrecovery.go:55","time":"2017-02-24T03:34:27Z"}

[brian-brazil]

That's an older version, my memory says we fixed that one. Try 1.5.2

[mg03]

ok. Thanx

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.