Target Scrape Timing Issue (Possibly Related to golang/go#16579)

[paulcavallaro]

What did you do?

Downloaded and ran default prometheus 1.1.3 darwin-amd64 with the default configuration. Note I also tried the whole thing with prometheus 0.20.0 darwin-amd64 as well.

What did you expect to see?

I watched http://localhost:9090/targets and expected the last scrape time to update every 5 seconds, but it did not.

What did you see instead? Under which circumstances?

It performed one scrape and then never scraped again.

Environment

MacOS Sierra.

Very likely related to: golang/go#16579

• System information:

  Darwin 16.0.0 x86_64

• Prometheus version:

  prometheus, version 1.1.3 (branch: master, revision: ac374aa)
     build user:       root@3e392b8b8b44
     build date:       20160916-11:39:10
     go version:       go1.6.3

[juliusv]

@paulcavallaro Thanks for the report! I don't have a Mac to reproduce. Any chance you could build Prometheus (just make build) with the tip version of Go and confirm that the problem goes away with that?

[paulcavallaro]

I'll have to double check later, but I believe my co-worker did that, running MacOS Sierra and was able to get it working building from prometheus head + golang 1.7.

[fiadliel]

I just downloaded the 1.2.0 release for OSX from the website, and it crashed regularly and quickly from browsing the HTTP interface (with the default OOTB configuration). Recompiling with the latest go compiler was fine; it looks like your current releases are quite broken on OSX Sierra.

(edit: sorry, never mind: it was 1.1.3 I downloaded, earlier today, so I assume you've updated your go toolchain for that release)

(edit: nope, it still crashes immediately after start, if I browse the website:
INFO[0000] Starting target manager... source=targetmanager.go:76
unexpected fault address 0xb01dfacedebac1e
fatal error: fault
[signal 0xb code=0x1 addr=0xb01dfacedebac1e pc=0x3e4e5d]
)

[grobie]

And it works fine if you compile it natively?

@sdurrheimer any ideas? Could it be related to the cgo changes?

[fiadliel]

Yes, the version compiled with Go 1.7.1 is running fine for the last hour or two, but refreshing the web page repeatedly causes the distributed version to die almost immediately.

I'm not that familiar with Go, but this goroutine stack always seems to be present:

goroutine 82 [running]:
runtime.throw(0xce8840, 0x5)
    /usr/local/go/src/runtime/panic.go:547 +0x90 fp=0xc8218837e8 sp=0xc8218837d0
runtime.sigpanic()
    /usr/local/go/src/runtime/sigpanic_unix.go:27 +0x2ba fp=0xc821883838 sp=0xc8218837e8
github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route.(*Router).handle.func1(0x192b930, 0xc820295e10, 0xc821ade000, 0x0, 0x0, 0x0)
    /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route/route.go:83 +0x55d fp=0xc8218839f8 sp=0xc821883838
github.com/prometheus/prometheus/vendor/github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc820013d00, 0x192b930, 0xc820295e10, 0xc821ade000)
    /go/src/github.com/prometheus/prometheus/vendor/github.com/julienschmidt/httprouter/router.go:299 +0x193 fp=0xc821883b38 sp=0xc8218839f8
github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route.(*Router).ServeHTTP(0xc82000b520, 0x192b930, 0xc820295e10, 0xc821ade000)
    /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route/route.go:125 +0x42 fp=0xc821883b60 sp=0xc821883b38
net/http.serverHandler.ServeHTTP(0xc8202ce000, 0x192b930, 0xc820295e10, 0xc821ade000)
    /usr/local/go/src/net/http/server.go:2081 +0x19e fp=0xc821883bc0 sp=0xc821883b60
net/http.(*conn).serve(0xc821870000)
    /usr/local/go/src/net/http/server.go:1472 +0xf2e fp=0xc821883f88 sp=0xc821883bc0
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc821883f90 sp=0xc821883f88
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2137 +0x44e

[sdurrheimer]

@grobie Prometheus doesn't use cgo since 1.1.3.

[beorn7]

golang/go#16579 ? Not sure how that would be connected, but it is something happening in MacOS Sierra and it is fixed in Go1.7.

BTW: Shouldn't we move our build pipeline to 1.7 finally?

[beorn7]

https://github.com/prometheus/prometheus/releases/tag/v1.2.1 is currently being built with Go1.7.1.Binaries should show up any second there. Please try them out.

[brian-brazil]

I'm going to presume this is resolved.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.