Prometheus Stable (0.14.0) segfault in local/storage.go

[mwitkow]

We're running:

prometheus, version 0.14.0 (branch: stable, revision: 67e7741)
  build user:       root
  build date:       20150603-06:20:33
  go version:       1.4.2

We got Prometheus segfaulting periodically on multiple instances. We do periodically SIGHUP to reload configs, and we do have of warnings of

time="2015-07-15T16:25:49Z" level=warning msg="Error expanding alert template OurMagicalAlert with data '{map[deployment_name:Foobar metric:Load severity:page] 13}': error parsing template __alert_OurMagicalAlert: template: __alert_OurMagicalAlert:1: function \"labels\" not defined" file=manager.go line=201

a couple of times over a window of 2-3 minutes before:

unexpected fault address 0xc2172e87b0
fatal error: fault
[signal 0xb code=0x2 addr=0xc2172e87b0 pc=0xc2172e87b0]

goroutine 81 [running]:
runtime.gothrow(0xb13be0, 0x5)
    /usr/lib/go/src/runtime/panic.go:503 +0x8e fp=0xc2172e85c0 sp=0xc2172e85a8
runtime.sigpanic()
    /usr/lib/go/src/runtime/sigpanic_unix.go:29 +0x265 fp=0xc2172e8610 sp=0xc2172e85c0
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start
    /go/src/github.com/prometheus/prometheus/storage/local/storage.go:240 +0x502

goroutine 1 [select, 19 minutes]:

We do have a full set of error stack available off-record.

Is this a known issue?

LOLz bonus: I posted this by mistake to fleet: coreos/fleet#1309 ;)

[fabxc]

This is not a known issue.

The rule file with the alert raising the warning and any additional error stacks would probably help a lot to track this down. Does the segfault always happen in storage.go:240?

[mwitkow]

@fabxc, I'm happy to drop them off-public record.

[fabxc]

@mwitkow-io fab.reinartz@gmail.com

[mwitkow]

So yea, it seems like it was an OS issue. Rolling back CoreOS from 735.0.0 to 723.0.0 made Prometheus stable again.

It seems that 735 was broken as per newer release note:
https://coreos.com/releases/#745.1.0

We experienced similar issues with Etcd and other Go programs. Apologies if we have wasted your time.

[fabxc]

No worries at all. Glad this could be resolved quickly.
Closing here then.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.