Prometheus 2.2.1 fails to parse certificate probably due to Go version.

[ffilippopoulos]

What did you do?
Upgraded for prometheus version from 2.1.0 to 2.2.1 on kubernetes cluster.

What did you expect to see?

Expected to see prometheus being able to identify and scrape all kubernetes targets since configuration was not changed.

What did you see instead? Under which circumstances?

Prometheus was unable to see anything related to kubernetes, as it was failing to parse the certificate coming from master.

Environment

• System information:

  Linux 4.14.19-coreos x86_64

• Prometheus version:

  version 2.2.1

• Prometheus config:
  Using config from the documentation https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml is enough to reproduce this

• Logs:

level=error ts=2018-03-19T11:14:16.374707313Z caller=main.go:216 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:354: Failed to list *v1.Node: Get https://10.3.0.1:443/api/v1/nodes?resourceVersion=0: tls: failed to parse certificate from server: x509: cannot parse dnsName \"x.x.x.x.x.x.\""

That should be happening because the address on my certificate contains a trailing dot (coming from aws that way and is compatible with specs about dns names).
Looks like this is a Golang related issue and a fix from their side is planned here: kubernetes/client-go#371, but I assumed that it should be tracked by prometheus as well as it is breaking functionality.

[dannyk81]

I've encountered the same issue, this is indeed due to the way tls works in Go 1.10.

Fortunately, golang/go@4b1d704 was merged into Golang master (details golang/go#23711) and I was able to build Prometheus using Go built from master.

This solved the issue for me.

[george-angel]

We would like to abstain from creating a custom Prometheus image and managing it, any chance of a patch release to fix this?

[brian-brazil]

I'm not sure it's wise to start doing releases with development versions of Go, so we should wait until Go releases this.

[dannyk81]

💯 I just wanted to confirm that this is fixed in Go, but not released yet.

[dannyk81]

Go 1.10.1 released with the fix.

https://groups.google.com/forum/m/#!msg/golang-announce/IkPkOF8JqLs/TFBbWHJYAwAJ

[beorn7]

As we are using go1.10.1 for new builds, can we consider this fixed?

[brian-brazil]

Yip. Whatever we release next will have the fix.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.