Marathon service discovery successful for 500 response with valid JSON in response body

[rohit01]

Problem statement

We run multiple Prometheus clusters at large scale with Marathon service discovery. At times, marathon responds with 500 error but a valid JSON response is returned. Marathon service discovery does not check for HTTP status code and continues to process. This results in a job without targets and services are not monitored until the next API call.

Use case. Why is this important?
We miss metrics intermittently.

Solution

Treat non-2xx HTTP response as error. It will preserve the existing targets.

Bug Report

What did you do?
It's happening on our production systems intermittently. Was able to reproduce the same with simulated marathon responses in staging.

What did you expect to see?
No target loss. No missing metrics.

What did you see instead? Under which circumstances?
Sudden metric drop for all containers at once triggering alerts.

• Prometheus version:
  Prometheus 2.2.1

• Alertmanager version:
  version 0.11.0

• Prometheus configuration file:
  A standard config with marathon_sd_config

• Logs:
  All ok in logs.

[rohit01]

Raised a pull request for fixing this issue: #4091

@beorn7 @fabxc @xperimental: Please take a look and let me know if you want any changes before merging it to master.

Thanks for your time and contributions :)

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.