Debug logs for failed scrapes

[discordianfish]

Hi,

I need to debug intermediate scrape failures and realize there is no way to differentiate between various reasons for a scrape failure after the fact. I can make an educated guess whether a scrape timed out or failed otherwise by using scrape_duration_seconds but I can't differentiate between DNS issues, connection refused, resetted connections, EOF and so on.

Therefor I propose we log scrape failures. I'd propose at info level severity. I consider a scrape failure important enough to not have to enable debug logging, especially given that info is already quite talky with the maintenance sweep messages.

Alternatively, which would work for me as much, we could try to come up with a timeseries or additional labels for up, but I think people would prefer logging.

[brian-brazil]

Scrape failures is roughly speaking request logging, it doesn't belong in application logs and could get extremely spammy (potentially hundreds of lines per second). Debug would be the appropriate log level.

[discordianfish]

For context: On my cluster the sweep messages are pretty spamming (every 10-15 seconds) which lead me to suggest info log level.
While I'd still prefer not to enable debug logging to get scrape failures, I'd be okay with that.

[juliusv]

I agree that debug would be appropriate level here, as it's not a concern of the health of the Prometheus server itself and can indeed get very noisy in larger Prometheus servers.

You can't just put additional error labels on up, as that'd break any query that wants to look at the up time series of a given target over time (because it would become multiple series).

[gouthamve]

For those looking to take this up, the relevant code path is here: https://github.com/prometheus/prometheus/blob/master/retrieval/scrape.go#L303 You could check each instance where an error is being produced and log accordingly.

[brian-brazil]

This change should probably be made in the dev-2.0 branch, as this code path has changed considerably.

https://github.com/prometheus/prometheus/blob/dev-2.0/retrieval/scrape.go#L388

[cstyan]

AFAICT we don't have access to the logger from the scrape function, so debug logging of scrape errors could happen in this block: https://github.com/prometheus/prometheus/blob/dev-2.0/retrieval/scrape.go#L638-L642

Am I missing anything?

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.