federate target error with net/http request canceled

[guanglinlv]

hi, my prometheus server will federate metrics from the other prometheus. net/http request canceled occurs after a day of running.

• prometheus runtime information

I built the binary from master yesterday.

• prometheus targets

• http_request_duration_microseconds{handler="federate"}

• no related errors in prometheus log
• i had checked the federate target's response is about 64 MegaBytes. is there a option to enable gzip?

i'm using the prometheus-.17.0 all the time before updating to 0.19.1, the federation always work correctly. so,i will roll back to 0.17.0 and test it again.

any change on it or any suggestion? thanks a lot.

[beorn7]

Pre-0.19 had a bug that would not honor the set timeout properly. Your error message looks like you are hitting the timeout while retrieving metrics.

Federation can sometimes be somewhat slow (but was sped up a lot with 0.18). In your case, I'd try to increase the timeout configured for the federation target (and increase the scrape interval if needed, the latter cannot be smaller than the former).

[guanglinlv]

@beorn7 thanks for your reply.

yes, it is a little slow to federate metrics from the other one. if i enlarge the federation interval, whether i will lose some metrics at some timestamp?

take for example:

• prometheus B scrap metrics from exporters every 5 seconds.
• prometheus A federate metrics from prometheus B every 30 seconds.

is it means that prometheus A will miss 25 seconds data ?

thanks.

[beorn7]

The resolution on your federating server will then indeed be lower than the resolution on the server you are federating from. Some users deliberately use this setup for downsampling.
No averaging is applied, you will always federate the latest possible value of each series.

[guanglinlv]

the samples of the federate targets is too large to get back in time. thanks. closing it.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.