prometheus-to-sd: Unknown metric error logged every minute

[evanj]

I'm running two clusters that were upgraded to 1.8.3-gke.0, and are running gcr.io/google-containers/prometheus-to-sd:v0.2.2. Every minute, this container logs:

Error while sending request to Stackdriver googleapi: Error 400: Unknown metric: container.googleapis.com/internal/addons/event_exported/stackdriver_sink_received_entry_count, badRequest

It seems like this must be some sort of mismatch between the expected names, and the actual names in Stackdriver or something?

[crassirostris]

@loburm Please take a look

[jpalomaki]

I can also see this for a GKE cluster running 1.8.3-gke.0

[loburm]

This one was merged into master, 1.7 and 1.8 branch. So with new release everything should work.

Sorry for this stupid typo.

[cwenzelg]

I'm still getting this error with 1.8.4-gke.0 and 1.8.4-gke.1 . Is there something else wrong or do i need to do something after updating the version?

Error while sending request to Stackdriver googleapi: Error 400: Unknown metric: container.googleapis.com/internal/addons/event_exported/stackdriver_sink_received_entry_count, badRequest

[x13n]

1.8.4 doesn't have this fix, it was added in 1.8.5.

[psaia]

Looks like this one may have come back from the dead (or something similar):

Error while sending request to Stackdriver googleapi: Error 400: Field timeSeries[0].metric.labels[0] had an invalid value of "grpc": Unrecognized metric label., badRequest

I'm getting that every minute or so.

[x13n]

This looks like #151