Skip to content

/prometheus

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

panic: runtime error: invalid memory address or nil pointer dereference #1703

Closed
wkshare opened this Issue Jun 3, 2016 · 9 comments

Comments

Assignees
No one assigned
Labels
kind/bug
Projects
None yet
Milestone
No milestone
6 participants
@wkshare @fabxc @juliusv @wkruse @slene @brian-brazil
@wkshare
Copy link

wkshare commented Jun 3, 2016 •
edited by brian-brazil

Can not start prometheus

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x28 pc=0x720455]

goroutine 262 [running]:
panic(0xd864c0, 0xc8200100d0)
    /usr/local/go/src/runtime/panic.go:481 +0x3e6
github.com/prometheus/prometheus/vendor/golang.org/x/net/context.propagateCancel(0x0, 0x0, 0x7fc937f05660, 0xc822fb0680)
    /go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/context/pre_go17.go:80 +0x25
github.com/prometheus/prometheus/vendor/golang.org/x/net/context.WithCancel(0x0, 0x0, 0x0, 0x0, 0xc820ae79a8)
    /go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/context/pre_go17.go:66 +0x126
github.com/prometheus/prometheus/retrieval.newScrapeLoop(0x0, 0x0, 0x7fc937ec0998, 0xc822fb83f0, 0x7fc937ec0968, 0xc82085c340, 0x7fc937ec0968, 0xc82085c360, 0x0, 0x0)
    /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:380 +0x124
github.com/prometheus/prometheus/retrieval.(*scrapePool).sync(0xc822fc4480, 0xc820a5a0a0, 0x1, 0x1)
    /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:228 +0x36d
github.com/prometheus/prometheus/retrieval.(*targetSet).sync(0xc822fc0190)
    /go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:258 +0x239
github.com/prometheus/prometheus/retrieval.(*targetSet).runProviders(0xc822fc0190, 0x7fc937f05690, 0xc820a56840, 0xc822fb6960)
    /go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:334 +0x34b
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload(0xc8202ba9a0)
    /go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:124 +0x25e
github.com/prometheus/prometheus/retrieval.(*TargetManager).Run(0xc8202ba9a0)
    /go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:79 +0x1cf
created by main.Main
    /go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:172 +0x171f
@fabxc

This comment has been minimized.

Sign in to view
Copy link
Member

fabxc commented Jun 3, 2016

Can you tell us the build information (logged on startup), the general logs up to this point and your configuration?
@wkshare

This comment has been minimized.

Sign in to view
Copy link
Author

wkshare commented Jun 3, 2016

I downloaded the binary package
9.76 MB prometheus-0.19.2.linux-amd64.tar.gz
@juliusv

This comment has been minimized.

Sign in to view
Copy link
Member

juliusv commented Jun 5, 2016

@wkshare and configuration and generated logs until the crash?

@brian-brazil brian-brazil added the kind/bug label Jun 9, 2016

@wkruse

This comment has been minimized.

Sign in to view
Copy link

wkruse commented Jun 10, 2016

Got it also. The generated logs until the crash:

time="2016-06-10T13:06:29Z" level=info msg="Starting prometheus (version=0.19.2, branch=master, revision=23ca13c)" source="main.go:73"
time="2016-06-10T13:06:29Z" level=info msg="Build context (go=go1.6.2, user=root@134dc6bbc274, date=20160529-18:58:00)" source="main.go:74"
time="2016-06-10T13:06:29Z" level=info msg="Loading configuration file /etc/prometheus/prometheus.yml" source="main.go:206"
time="2016-06-10T13:06:29Z" level=info msg="Loading series map and head chunks..." source="storage.go:340"
time="2016-06-10T13:06:29Z" level=info msg="0 series loaded." source="storage.go:345"
time="2016-06-10T13:06:29Z" level=info msg="Listening on :9090" source="web.go:241"
time="2016-06-10T13:06:29Z" level=info msg="Starting target manager..." source="targetmanager.go:74"
time="2016-06-10T13:06:29Z" level=warning msg="No AlertManager configured, not dispatching any alerts" source="notifier.go:165"

Running official Docker image:

prom/prometheus                                       0.19.2               fbc0b6ce5d5c        11 days ago         43.18 MB
@wkruse

This comment has been minimized.

Sign in to view
Copy link

wkruse commented Jun 10, 2016

I was running Docker Client Version: 1.11.2 and Docker Server Version: 1.11.1. Upgrading Docker Server to Version: 1.11.2 fixed it for me.
@slene

This comment has been minimized.

Sign in to view
Copy link

slene commented Jun 14, 2016

Also has this problem, make me crazy.

INFO[0000] Starting prometheus (version=0.19.2, branch=master, revision=05b918a)  source=main.go:73
INFO[0000] Build context (go=go1.6.2, user=root@e19bc1d10c15, date=20160607-16:25:02)  source=main.go:74
INFO[0000] Loading configuration file prometheus.yml     source=main.go:206
INFO[0000] Loading series map and head chunks...         source=storage.go:341
INFO[0024] 1867719 series loaded.                        source=storage.go:346
WARN[0024] No AlertManagers configured, not dispatching any alerts  source=notifier.go:174
INFO[0024] Starting target manager...                    source=targetmanager.go:74
INFO[0024] Listening on :9090                            source=web.go:241
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x28 pc=0x723e05]

goroutine 202 [running]:
panic(0xd88cc0, 0xc82000e0f0)
    /usr/local/go/src/runtime/panic.go:481 +0x3e6
github.com/prometheus/prometheus/vendor/golang.org/x/net/context.propagateCancel(0x0, 0x0, 0x7ff3f7bb05c0, 0xc944c1f440)
    /go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/context/pre_go17.go:80 +0x25
github.com/prometheus/prometheus/vendor/golang.org/x/net/context.WithCancel(0x0, 0x0, 0x0, 0x0, 0xc94217b9a8)
    /go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/context/pre_go17.go:66 +0x126
github.com/prometheus/prometheus/retrieval.newScrapeLoop(0x0, 0x0, 0x7ff3f7b81058, 0xc944c42880, 0x7ff3f7b81028, 0xc90c534580, 0x7ff3f7b81028, 0xc90c5345a0, 0x0, 0x0)
    /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:380 +0x124
github.com/prometheus/prometheus/retrieval.(*scrapePool).sync(0xc944c52c00, 0xc820e55c18, 0x1, 0x1)
    /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:228 +0x36d
github.com/prometheus/prometheus/retrieval.(*targetSet).sync(0xc9076f29b0)
    /go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:258 +0x239
github.com/prometheus/prometheus/retrieval.(*targetSet).runProviders(0xc9076f29b0, 0x7ff3f7bb05f0, 0xc94218e0c0, 0xc8ed919530)
    /go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:334 +0x34b
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload(0xc8202bf3b0)
    /go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:124 +0x25e
github.com/prometheus/prometheus/retrieval.(*TargetManager).Run(0xc8202bf3b0)
    /go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:79 +0x1cf
created by main.Main
    /go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:172 +0x171f
@fabxc

This comment has been minimized.

Sign in to view
Copy link
Member

fabxc commented Jun 14, 2016

@slene Does it happen directly on startup. Can you show us your configuration?

fabxc added a commit that referenced this issue Jun 14, 2016

@fabxc
retrieval: don't sync to uninitialized scrape pool 
This change does just signal a scrape target update to the scraping loop
once an initial target set is fetched.
Before, the scrape pool was directly synced, causing a race against an
uninitialized scrape pool.

Fixes #1703
Loading status checks…
d0eeae9

@fabxc fabxc referenced this issue Jun 14, 2016

Merged

retrieval: don't sync to uninitialized scrape pool #1736

@fabxc

This comment has been minimized.

Sign in to view
Copy link
Member

fabxc commented Jun 14, 2016

@slene @wkshare feel free to test it again with #1736. This will be part of 0.19.3, which we'll try to release today.

@fabxc fabxc closed this in #1736 Jun 14, 2016

fabxc added a commit that referenced this issue Jun 14, 2016

@fabxc
retrieval: don't sync to uninitialized scrape pool 
This change does just signal a scrape target update to the scraping loop
once an initial target set is fetched.
Before, the scrape pool was directly synced, causing a race against an
uninitialized scrape pool.

Fixes #1703
4c864c8
@lock

This comment has been minimized.

Sign in to view
Copy link

lock bot commented Mar 24, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Mar 24, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.