Multiple K8s SD instances causing scrape issues.

[tcolgate]

What did you do?

Configured multiple instances of kube sd (9 in total all small clusters)

What did you expect to see?

normal performance

What did you see instead? Under which circumstances?

Target scrapes mostly stay in UNKNOWN or flit in / out

• Prometheus version:

1.1.3 from dockerhub latest

• Prometheus configuration file:

may provide offline

• Logs:

Probably related, we are seeing

time="2016-09-22T13:45:09Z" level=error msg="Watch services unexpectedly closed: EOF" source="service.go:188"
time="2016-09-22T13:45:20Z" level=error msg="Watch services unexpectedly closed: EOF" source="service.go:188"
time="2016-09-22T13:45:50Z" level=error msg="Watch nodes unexpectedly closed: EOF" source="node.go:162"
time="2016-09-22T13:46:22Z" level=error msg="Watch service endpoints unexpectedly closed: EOF" source="service.go:228"
time="2016-09-22T13:46:36Z" level=error msg="Watch services unexpectedly closed: EOF" source="service.go:188"
time="2016-09-22T13:46:58Z" level=error msg="Watch services unexpectedly closed: EOF" source="service.go:188"
time="2016-09-22T13:47:03Z" level=error msg="Watch services unexpectedly closed: EOF" source="service.go:188"
time="2016-09-22T13:47:20Z" level=error msg="Watch pods unexpectedly closed: EOF" source="pod.go:169"

We also see this on another instances which is not manifesting problems

[tcolgate]

Flame graph
torch.svg.gz

[tcolgate]

sum(rate(prometheus_target_scrape_pool_sync_total[1m])) shows > 1.5 continuously,
rate(prometheus_target_skipped_scrapes_total[1m]) > 6

[tcolgate]

The problems may be unrelated to the kube SD, I think the memory chunks settings have been misconfigured.

[tcolgate]

This had nothing to do with the service discovery, and was down to a misconfiguration.

[shamil]

@tcolgate I'm hitting this too...
can you give more information, what you did in configuration to solve this issue?

[shamil]

@tcolgate , never mind, this was caused by #2068. Upgrading to 1.2.1 solved this

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.