SIGSEGV on startup (armv7, prometheus-2.0.0-beta.2)

[marcan]

Prometheus crashes almost immediately on startup, seemingly after creating the first tsdb block. This is prometheus 2.0.0-beta.2 on a Raspberry Pi 3 (armv7)

Environment

• System information:
  Linux 4.9.39-1-ARCH armv7l

• Prometheus version:

prometheus, version 2.0.0-beta.2 (branch: HEAD, revision: a52f082939a566d5269671e98be06fc6bdf61d09)
  build user:       root@41a0740ea598
  build date:       20170818-08:48:32
  go version:       go1.8.3

• Prometheus configuration file:

global:
  scrape_interval:     15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets:
        - 'localhost:9090'

• Logs:

# /usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus/data/ --storage.tsdb.retention=192h
INFO[0000] Starting prometheus (version=2.0.0-beta.2, branch=HEAD, revision=a52f082939a566d5269671e98be06fc6bdf61d09)  source="main.go:204"
INFO[0000] Build context (go=go1.8.3, user=root@41a0740ea598, date=20170818-08:48:32)  source="main.go:205"
INFO[0000] Host details (Linux 4.9.39-1-ARCH #1 SMP Sat Jul 22 16:54:53 UTC 2017 armv7l hmc02 (none))  source="main.go:206"
INFO[0000] Starting tsdb                                 source="main.go:218"
INFO[0000] tsdb started                                  source="main.go:224"
INFO[0000] Loading configuration file /etc/prometheus/prometheus.yml  source="main.go:357"
INFO[0000] Server is Ready to receive requests.          source="main.go:334"
INFO[0000] Listening on 0.0.0.0:9090                     source="web.go:327"
INFO[0000] Starting target manager...                    source="targetmanager.go:67"
ts=2017-09-04T20:35:36.727638381Z caller=db.go:901 msg="created head block" ulid=01BS79P24ARWY0P6K081FBGVTG mint=1504548000000 maxt=1504555200000
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x11580]

goroutine 164 [running]:
sync/atomic.loadUint64(0x122c6094, 0x4e0c9100, 0x15e)
        /usr/local/go/src/sync/atomic/64bit_arm.go:10 +0x3c
github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*HeadBlock).Meta(0x122c6000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:219 +0x98
github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).ensureHead(0x1230c990, 0x4e9b0859, 0x15e, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:771 +0x1a8
github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*dbAppender).appenderAt(0x125941e0, 0x4e9b0859, 0x15e, 0x0, 0x0, 0x1eda4)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:709 +0x11c
github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*dbAppender).Add(0x125941e0, 0x122d0e00, 0x4, 0x4, 0x4e9b0859, 0x15e, 0x900d762d, 0x3f243bee, 0x1, 0x13a46e4, ...)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:655 +0x2c
github.com/prometheus/prometheus/storage/tsdb.appender.Add(0x1eade78, 0x125941e0, 0x122d0e00, 0x4, 0x4, 0x4e9b0859, 0x15e, 0x900d762d, 0x3f243bee, 0x126cc460, ...)
        /go/src/github.com/prometheus/prometheus/storage/tsdb/tsdb.go:133 +0x68
github.com/prometheus/prometheus/storage/tsdb.(*appender).Add(0x125e41e0, 0x122d0e00, 0x4, 0x4, 0x4e9b0859, 0x15e, 0x900d762d, 0x3f243bee, 0x2, 0x50, ...)
        <autogenerated>:7 +0x84
github.com/prometheus/prometheus/storage.(*fanoutAppender).Add(0x12594200, 0x122d0e00, 0x4, 0x4, 0x4e9b0859, 0x15e, 0x900d762d, 0x3f243bee, 0x1257cd70, 0x122d0e00, ...)
        /go/src/github.com/prometheus/prometheus/storage/fanout.go:105 +0x58
github.com/prometheus/prometheus/retrieval.(*timeLimitAppender).Add(0x126cc410, 0x122d0e00, 0x4, 0x4, 0x4e9b0859, 0x15e, 0x900d762d, 0x3f243bee, 0x0, 0x0, ...)
        /go/src/github.com/prometheus/prometheus/retrieval/target.go:239 +0x94
github.com/prometheus/prometheus/retrieval.ruleLabelsAppender.Add(0x1ead9f0, 0x126cc410, 0x1257cc80, 0x2, 0x5, 0x12594260, 0x2, 0x2, 0x4e9b0859, 0x15e, ...)
        /go/src/github.com/prometheus/prometheus/retrieval/target.go:274 +0x228
github.com/prometheus/prometheus/retrieval.(*ruleLabelsAppender).Add(0x12594220, 0x12594260, 0x2, 0x2, 0x4e9b0859, 0x15e, 0x900d762d, 0x3f243bee, 0x12400cc8, 0x8, ...)
        <autogenerated>:52 +0x7c
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append(0x122d0a80, 0x12ca6000, 0x9f2b, 0x10000, 0xd13fb018, 0xe, 0x27b16a01, 0x1f5d018, 0x1eb08, 0x1f480, ...)
        /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:830 +0x374
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0x122d0a80, 0x7e11d600, 0x3, 0x540be400, 0x2, 0x0)
        /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:646 +0x3cc
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
        /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:302 +0x468

[marcan]

This looks like another instance of #2666 (in a different part of the code).

[fabxc]

This particular code path no longer exists in beta.3, so I'm closing here.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.