Metrics not updated on external probe timeout

[ltagliamonte-dd]

We use cloudprober with an external probe that executes a check on consul.
When the external probe hits a timeout the prometheus metric total for that probe doesn't get increments.

In the logs I can clearly see that the external probe hits timeout:

E0823 19:30:51.106972      74 external.go:482] [cloudprober.consul_probe/consul_us_west_2] context deadline exceeded

But the Prometheus metric doesn't get any increments:

total{ptype="external",probe="consul_probe/consul_us_west_2",dst=""} 

i'm making the probe failing with an iptables rule (can't connect to consul for the check)

as soon as I make the probe pass again (removing the iptables rule) the prometheus total counter start updating again with the correct number (will report also the number of check failed)

The problem with this behavior is that if the probe keeps timing out the configured alert will never trigger because the metric will never update and my alert is configured on the diff between total and success counters.

currently we are using cloudprober@v0.11.0 i didn't find any existing issue related to this behavior.

Flat line in the monitoring system shows this behavior as well:

[manugarg]

Thanks for the bug report, @ltagliamonte-dd. "total" should increment if there is a timeout. I'll look into it.

[manugarg]

Problem seems to be here:

cloudprober/probes/external/external.go

Line 501 in 30fb6ab

p.processProbeResult(&probeStatus{

We do results processing in the goroutine that gets canceled on timeout. We should move this out.

[manugarg]

@ltagliamonte-dd This should now be fixed in the "master" branch (master tag). If you can verify with the latest changes, it will be great. I'll try to cut the next release (milestone: v0.11.3) in a week or so.

[manugarg]

Closing this issue now. Please feel free to reopen if it doesn't work. You can track the release here:
https://github.com/google/cloudprober/milestone/11

[ltagliamonte]

Thanks for the quick resolution, with test it out and reopen if necessary.

[ltagliamonte-dd]

Thank you @manugarg i was able to re-test and it looks like the issue has been fixed.