Crash when missing TLS certificate files

[matthiasr]

I got the following crash when adding a new job with a tls_config and the expected certificate files weren't there:

2016-02-08_17:43:07.35821 time="2016-02-08T17:43:07Z" level=warning msg="Crash recovery complete." source="crashrecovery.go:143"
2016-02-08_17:43:07.35932 time="2016-02-08T17:43:07Z" level=info msg="403149 series loaded." source="storage.go:273"
2016-02-08_17:43:07.35954 time="2016-02-08T17:43:07Z" level=info msg="Listening on :9090" source="web.go:220"
2016-02-08_17:43:07.35964 time="2016-02-08T17:43:07Z" level=info msg="Starting target manager..." source="targetmanager.go:114"
2016-02-08_17:43:07.37987 time="2016-02-08T17:43:07Z" level=error msg="cannot create HTTP client: unable to use specified client cert (/srv/prometheus/k8s-certificates/client.crt) & key (/srv/prometheus/k8s-certificates/key.crt): open /srv/prometheus/k8s-certificates/key.crt: no such file or directory" source="target.go:208"
2016-02-08_17:43:07.38006 time="2016-02-08T17:43:07Z" level=error msg="cannot create HTTP client: unable to use specified client cert (/srv/prometheus/k8s-certificates/client.crt) & key (/srv/prometheus/k8s-certificates/key.crt): open /srv/prometheus/k8s-certificates/key.crt: no such file or directory" source="target.go:208"
2016-02-08_17:43:07.38025 time="2016-02-08T17:43:07Z" level=error msg="cannot create HTTP client: unable to use specified client cert (/srv/prometheus/k8s-certificates/client.crt) & key (/srv/prometheus/k8s-certificates/key.crt): open /srv/prometheus/k8s-certificates/key.crt: no such file or directory" source="target.go:208"
2016-02-08_17:43:07.38370 panic: non-positive interval for NewTicker
2016-02-08_17:43:07.38371
2016-02-08_17:43:07.38372 goroutine 307 [running]:
2016-02-08_17:43:07.38372 time.NewTicker(0x0, 0xc8600bc700)
2016-02-08_17:43:07.38372       /usr/local/opt/go/libexec/src/time/tick.go:23 +0xe4
2016-02-08_17:43:07.38374 github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc860078870, 0x7f13e9c7d568, 0xc8200f7d00)
2016-02-08_17:43:07.38374       /Users/fabianreinartz/repos/src/github.com/prometheus/prometheus/retrieval/target.go:324 +0x22a
2016-02-08_17:43:07.38374 created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
2016-02-08_17:43:07.38375       /Users/fabianreinartz/repos/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:302 +0x780
2016-02-08_17:43:07.38375
2016-02-08_17:43:07.38375 goroutine 1 [select]:
2016-02-08_17:43:07.38376 main.Main(0x0)
2016-02-08_17:43:07.38376       /Users/fabianreinartz/repos/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:176 +0x1ae6
2016-02-08_17:43:07.38377 main.main()
2016-02-08_17:43:07.38378       /Users/fabianreinartz/repos/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:46 +0x18
2016-02-08_17:43:07.38378
2016-02-08_17:43:07.38378 goroutine 105 [select]:
2016-02-08_17:43:07.38379 github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc82009ac60)
2016-02-08_17:43:07.38380       /Users/fabianreinartz/repos/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x54a
2016-02-08_17:43:07.38380 created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
2016-02-08_17:43:07.38380       /Users/fabianreinartz/repos/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x77f
2016-02-08_17:43:07.38381

[matthiasr]

Actually, what's missing is the key file, for which I mixed up the name.

[matthiasr]

And with a readable client key file it works again. Before, this was completely reproducible (i.e. it crashlooped).

[juliusv]

The problem seems to be that for startup target initialization, the same target updating code is used as later on, and when something is wrong with the target config options, this method just logs and error and returns, leaving behind a not properly initialized target:

https://github.com/prometheus/prometheus/blob/0.16.2/retrieval/target.go#L208

[juliusv]

Doing a fix...

[juliusv]

Fix PR in #1379

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.