healthchecks : Retry monitoringPing when InvalidArgument error.

[franciscovalentecastro]

Description

The API Check from the healthchecks package returns an InvalidArgument error when the Ops Agent is restarted very quickly :

[API Check] Result: ERROR, Detail: rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: One or more points were written more frequently than the maximum sampling period configured for the metric.

This is due to the maximum rate of "one point each 5 seconds" at which data can be written to a single time series. This PR retries monitoringPing 2 times with a 6 second backoff to avoid going above the rate.

Related issue

b/291631906

How has this been tested?

Checklist:

• Unit tests
  • Unit tests do not apply.
  • Unit tests have been added/modified and passed for this PR.
• Integration tests
  • Integration tests do not apply.
  • Integration tests have been added/modified and passed for this PR.
• Documentation
  • This PR introduces no user visible changes.
  • This PR introduces user visible changes and the corresponding documentation change has been made.
• Minor version bump
  • This PR introduces no new features.
  • This PR introduces new features, and there is a separate PR to bump the minor version since the last release already.
  • This PR bumps the version.

[avilevy18]

Do we want to consider matching the description as well as the code?

[franciscovalentecastro]

InvalidArgument can appear in a malformed metric write request. But in this case, the monitoringPing is generated to be correct. So it may have value checking for the specific error One or more points were written more frequently ... messaged.

The error APIError has the method Reason() that could match the text. But I'm unsure if that would be maintainable if the message changes in the future. We are also retrying once, so this won't extend long in an edge case.