Handle in-progress scrapes more gracefully on config reload

[SuperQ]

What did you do?
Reload (HUP) prometheus server to add new rules. In-flight scrapes get canceled with "context canceled". Thus making the up metric bounce to 0.

What did you expect to see?
Reload is graceful, and waits for in-flight scrapes to complete or timeout before continuing with reload.

What did you see instead? Under which circumstances?
The Prometheus server stops scheduling new scrapes, waits for the in-flight scrapes to complete or timeout, then reload the config.

Environment

• System information:

  insert output of uname -srm here

• Prometheus version:

  insert output of prometheus -version here

• Alertmanager version:

  insert output of alertmanager -version here (if relevant to the issue)

• Prometheus configuration file:

insert configuration here

• Alertmanager configuration file:

insert configuration here (if relevant to the issue)

• Logs:

insert Prometheus and Alertmanager logs relevant to the issue here

[SuperQ]

Another possible option would be to have a reload-rules action that doesn't update the targets, but only reloads the rule evaluation.

[brian-brazil]

The Prometheus server stops scheduling new scrapes, waits for the in-flight scrapes to complete or timeout, then reload the config.

Stopping all processing for potentially minutes isn't something we should do imho.

[fabxc]

Another possible option would be to have a reload-rules action that doesn't update the targets, but only reloads the rule evaluation.

Doesn't need a separate action but just a validation whether the configuration changed or not. Sounds useful in general. Want to take a stab at implementing that?

[brian-brazil]

We've had it in the past where a HUP was used to work around various SD stuckness, I'm not sure we should lose that.

[brian-brazil]

Closing in favour of #2756

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.