Panic

[stamm]

What did you do?
Nothing really

What did you expect to see?
No errors

What did you see instead? Under which circumstances?
Go panic error:

Environment

• System information:

Linux 3.10.0-327.10.1.el7.x86_64 x86_64

• Prometheus version:

With some patches for etcd

prometheus, version 1.2.1 (branch: HEAD, revision: c53b96b07da98e5d02be9364d137493a476ad89a)
build user: root@****
build date: 20161011-14:42:12
go version: go1.6.2

• Prometheus configuration file:

global: 
  scrape_interval: "10s"
  scrape_timeout: "9s"
  evaluation_interval: "15s"
rule_files: 
  - /etc/prometheus/generated-rules.rule
  - /etc/prometheus/rules/container_df.rule
  - /etc/prometheus/rules/container_mem_usage_bytes.rule
  - /etc/prometheus/rules/container_cpu_usage_5m_seconds.rule
scrape_configs: 
  - job_name: prometheus
    static_configs: 
      - targets: 
          - "localhost:9090"
  - job_name: goapi
    scrape_interval: "20s"
    etcd_sd_configs: 
      - endpoints: 
          - "http://etcd1:2379"
          - "http://etcd2:2379"
          - "http://etcd3:2379"
          - "http://etcd4:2379"
          - "http://etcd5:2379"
        key_prefix: /live
  - job_name: mysql-high
    scrape_interval: "1s"
    scrape_timeout: "1s"
    relabel_configs: 
      - target_label: job
        replacement: mysql
      - source_labels: 
          - "__address__"
        action: replace
        target_label: alias
        regex: "(.+):(\x5cd+)"
        replacement: "$1"
    file_sd_configs: 
      - files: 
          - /etc/prometheus/services/mysql-high.yaml
  - job_name: mysql-medium
    scrape_interval: "5s"
    scrape_timeout: "1s"
    relabel_configs: 
      - target_label: job
        replacement: mysql
      - source_labels: 
          - "__address__"
        action: replace
        target_label: alias
        regex: "(.+):(\x5cd+)"
        replacement: "$1"
    file_sd_configs: 
      - files: 
          - /etc/prometheus/services/mysql-medium.yaml
  - job_name: mysql-low
    scrape_interval: "60s"
    scrape_timeout: "5s"
    relabel_configs: 
      - target_label: job
        replacement: mysql
      - source_labels: 
          - "__address__"
        action: replace
        target_label: alias
        regex: "(.+):(\x5cd+)"
        replacement: "$1"
    file_sd_configs: 
      - files: 
          - /etc/prometheus/services/mysql-low.yaml
  - job_name: etcd
    scrape_interval: "20s"
    file_sd_configs: 
      - files: 
          - /etc/prometheus/services/etcd.yaml
  - job_name: cadvisor
    scrape_interval: "20s"
    file_sd_configs: 
      - files: 
          - /etc/prometheus/services/cadvisor.yaml
  - job_name: telegraf
    scrape_interval: "20s"
    file_sd_configs: 
      - files: 
          - /etc/prometheus/services/telegraf.yaml
  - job_name: elasticsearch
    file_sd_configs: 
      - files: 
          - /etc/prometheus/services/elasticsearch.yaml

• Logs:

2016-10-25T02:36:13.916504+00:00 prometheus prometheus: time="2016-10-25T02:36:13Z" level=error msg="http: panic serving 127.0.0.1:54558: runtime error: makeslice: cap out of range
goroutine 88087210 [running]:
net/http.(*conn).serve.func1(0xcce3357980)
/usr/lib/go-1.6/src/net/http/server.go:1389 +0xc1
panic(0x117be60, 0xce65f462f0)
/usr/lib/go-1.6/src/runtime/panic.go:443 +0x4e9
github.com/prometheus/prometheus/storage/local.(*memorySeries).preloadChunksForRange(0xc8d448a150, 0x80a0333b2855c6ac, 0x157f70f7d50, 0x157f52fc0d0, 0xc820245b80, 0x0, 0x0, 0x0, 0x0)
/builds/ebuilds/prometheus/src/github.com/ prometheus/prometheus/storage/local/series.go:500 +0x614
github.com/prometheus/prometheus/storage/local.(*MemorySeriesStorage).preloadChunksForRange(0xc820245b80, 0x80a0333b2855c6ac, 0xc8d448a150, 0x157f70f7d50, 0x157f52fc0d0, 0x0, 0x0)
/builds/ebuilds/prometheus/src/github.com/prometheus/prometheus/storage/local /storage.go:972 +0x141
github.com/prometheus/prometheus/storage/local.(*MemorySeriesStorage).QueryRange(0xc820245b80, 0x7fcdb2d027e8, 0xce0f2b9b00, 0x157f70f7d50, 0x157f52fc0d0, 0xc8664f0f00, 0x5, 0x8, 0x0, 0x0, ...)
/builds/ebuilds/prometheus/src/github.com/prometheus/prometheus/storage/local/storage.go:489 +0x2 0d
github.com/prometheus/prometheus/promql.(*Engine).populateIterators.func1(0x7fcdf2eead00, 0xcfc50ae050, 0xce4cc16798)
/builds/ebuilds/prometheus/src/github.com/prometheus/prometheus/promql/engine.go:495 +0x1bb
github.com/prometheus/prometheus/promql.inspector.Visit(0xce0f2b9b60, 0x7fcdf2eead00, 0xcfc50ae050, 0x0, 0x0)
/builds/ebuilds/prometheus/src/github.com/prometheus/prometheus/promql/ast.go:306 +0x40
github.com/prometheus/prometheus/promql.Walk(0x7fcdf2ebcf68, 0xce0f2b9b60, 0x7fcdf2eead00, 0xcfc50ae050)
/builds/ebuilds/prometheus/src/github.com/prometheus/prometheus/promql/ast.go:255 +0x71
github.com/promethe us/prometheus/promql.Walk(0x7fcdf2ebcf68, 0xce0f2b9b60, 0x7fcdb2d02850, 0xcc38705e00)
/builds/ebuilds/prometheus/src/github.com/prometheus/prometheus/promql/ast.go:275 +0x34a
github.com/prometheus/prometheus/promql.Walk(0x7fcdf2ebcf68, 0xce0f2b9b60, 0x7fcdf2eeaca8, 0xcc38705d40)
/builds/ebuilds/prometheus/src/github.com/prometheus/prometheus/promql/ast.go:285 +0xc1b
github.com/prometheus/prometheus/promql.Walk(0x7fcdf2ebcf68, 0xce0f2b9b60, 0x7fcd f2eead58, 0xcfc50ae0a0)
/builds/ebuilds/prometheus/src/github.com/prometheus/prometheus/promql/ast.go:278 +0x64f
github.com/prometheus/prometheus/promql.Walk(0x7fcdf2ebcf68, 0xce0f2b9b60, 0x7fcdf2eeaba0, 0xc8664f1100)
/builds/ebuilds/prometheus/src/github.com/prometheus/prometheus/promql/ast.go:281 +0x88d
git hub.com/prometheus/prometheus/promql.Inspect(0x7fcdf2eeaba0, 0xc8664f1100, 0xce0f2b9b60)
/builds/ebuilds/prometheus/src/github.com/prometheus/prometheus/promql/ast.go:316 +0x60
github.com/prometheus/prometheus/promql.(*Engine).populateIterators(0xc8202aaf20, 0x7fcdb2d027e8, 0xce0f2b9b00, 0xce0f2b9aa0, 0x0, 0x0)
 /builds/ebuilds/prometheus/src/github.com/prometheus/prometheus/promql/engine.go:501 +0x11e
github.com/prometheus/prometheus/promql.(*Engine).execEvalStmt(0xc8202aaf20, 0x7fcdb2d027e8, 0xce0f2b9b00, 0xc8664f1140, 0xce0f2b9aa0, 0x0, 0x0, 0x0, 0x0)
/builds/ebuilds/prometheus/src/github.com/prometheus/prometheus/p romql/engine.go:355 +0x124
github.com/prometheus/prometheus/promql.(*Engine).exec(0xc8202aaf20, 0x7fcdb2d027e8, 0xce0f2b9b00, 0xc8664f1140, 0x0, 0x0, 0x0, 0x0)
/builds/ebuilds/prometheus/src/github.com/prometheus/prometheus/promql/engine.go:344 +0x4b5
github.com/prometheus/prometheus/promql.(*query).Exec(0xc8664 f1140, 0x7fcdf2eded58, 0xc8664f0d80, 0x157f71067b0)
/builds/ebuilds/prometheus/src/github.com/prometheus/prometheus/promql/engine.go:196 +0x45
github.com/prometheus/prometheus/web/api/v1.(*API).queryRange(0xc8203468d0, 0xc9dca320e0, 0x0, 0x0, 0xffffffffffffffff)
/builds/ebuilds/prometheus/src/github.com/promet heus/prometheus/web/api/v1/api.go:207 +0x86f
github.com/prometheus/prometheus/web/api/v1.(*API).(github.com/prometheus/prometheus/web/api/v1.queryRange)-fm(0xc9dca320e0, 0x0, 0x0, 0x4)
/builds/ebuilds/prometheus/src/github.com/prometheus/prometheus/web/api/v1/api.go:126 +0x38
github.com/prometheus/prometheus/web/api/v1.(*API).Register.func1.1(0x7fcdd47a9830, 0xcc38705ce0, 0xc9dca320e0)
/builds/ebuilds/prometheus/src/github.com/prometheus/promethe us/web/api/v1/api.go:110 +0x4b
net/http.HandlerFunc.ServeHTTP(0xc8204df2e0, 0x7fcdd47a9830, 0xcc38705ce0, 0xc9dca320e0)
/usr/lib/go-1.6/src/net/http/server.go:1618 +0x3a
github.com/prometheus/prometheus/util/httputil.CompressionHandler.ServeHTTP(0x7fcdf2edd858, 0xc8204df2e0, 0x7fcdd47a9758, 0xccaac0c670, 0xc9dca 320e0)
/builds/ebuilds/prometheus/src/github.com/prometheus/prometheus/util/httputil/compression.go:90 +0x97
github.com/prometheus/prometheus/util/httputil.(*CompressionHandler).ServeHTTP(0xc8204df2f0, 0x7fcdd47a9758, 0xccaac0c670, 0xc9dca320e0)
<autogenerated>:5 +0xb6
net/http.(Handler).ServeHTTP-fm(0x7fcdd4 7a9758, 0xccaac0c670, 0xc9dca320e0)
/builds/ebuilds/prometheus/src/github.com/prometheus/prometheus/web/web.go:173 +0x50
github.com/prometheus/prometheus/vendor/github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x7fcdd47a9680, 0xccb71af040, 0xc9dca320e0)
/builds/ebuilds/prometh eus/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/client_golang/prometheus/http.go:178 +0x336
github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route.(*Router).handle.func1(0x7fcdd47a9680, 0xccb71af040, 0xc9dca320e0, 0x0, 0x0, 0x0)
/builds/ebuilds/prometheus/src/github.com/p rometheus/prometheus/vendor/github.com/prometheus/common/route/route.go:83 +0x55f
github.com/prometheus/prometheus/vendor/github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc820295f80, 0x7fcdd47a9680, 0xccb71af040, 0xc9dca320e0)
/builds/ebuilds/prometheus/src/github.com/prometheus/prometheus/vendor/github. com/julienschmidt/httprouter/router.go:299 +0x193
github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route.(*Router).ServeHTTP(0xc8202ab020, 0x7fcdd47a9680, 0xccb71af040, 0xc9dca320e0)
/builds/ebuilds/prometheus/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route/route.go:125 +0x42
net/http.serverHandler.ServeHTTP(0xc9ca9ac000, 0x7fcdd47a9680, 0xccb71af040, 0xc9dca320e0)
/usr/lib/go-1.6/src/net/http/se rver.go:2081 +0x19e
net/http.(*conn).serve(0xcce3357980)
/usr/lib/go-1.6/src/net/http/server.go:1472 +0xf2e
created by net/http.(*Server).Serve
/usr/lib/go-1.6/src/net/http/server.go:2137 +0x44e
" source="<autogenerated>:38"

[beorn7]

My guess would be that in https://github.com/prometheus/prometheus/blob/c53b96b07da98e5d02be9364d137493a476ad89a/storage/local/series.go#L500 , the calculated cap became negative. The most likely reason for that is data corruption that lead to non-monotonic timestamps in the series.

We should definitely guard against this by making sure the cap cannot become negative.

@stamm To verify that this is a data corruption problem, could you start from a blank storage and see if the problem recurs?

[stamm]

It's working fine right now, without even restarted

Active: active (running) since Sat 2016-10-22 10:57:51 UTC; 3 days ago

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.