federation failed with error message "write: broken pipe"

[ahrtr]

The prometheus version is v2.0.0. Previously it was working well. But I saw lots of the following errors today. I could still see the errors after rebooting the local prometheus instance.

level=error ts=2018-05-16T08:39:32.143676493Z caller=federate.go:163 component=web msg="federation failed" err="write tcp 192.168.243.145:9090->10.0.0.12:33494: write: broken pipe"
level=error ts=2018-05-16T08:40:32.146520927Z caller=federate.go:163 component=web msg="federation failed" err="write tcp 192.168.243.145:9090->10.0.0.12:33504: write: broken pipe"
level=error ts=2018-05-16T08:41:32.154047845Z caller=federate.go:163 component=web msg="federation failed" err="write tcp 192.168.243.145:9090->10.0.0.12:33506: write: broken pipe"
level=error ts=2018-05-16T08:42:32.145742427Z caller=federate.go:163 component=web msg="federation failed" err="write tcp 192.168.243.145:9090->10.0.0.12:33508: write: broken pipe"

[brian-brazil]

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[ervinb]

Note for future Googlers, as this is one of the first results when the error above is searched for.

This is caused by the misuse of federation. (how it should be used)

When large amounts of data is federated, the scrape_timeout is exceeded on the caller, which will cause it to hang up. As the connection is closed, the client runs into a write error - broken pipe - because the other end disconnected.

[caller - 1.2.3.4]
$ curl https://prom.abc.com/federate
# scrape_timeout amount of time passes without results -> hang up connection

[client log - 192.168.243.145]
level=error ts=2018-05-16T08:42:32.145742427Z caller=federate.go:163 component=web msg="federation failed" err="write tcp 192.168.243.145:9090->10.0.0.12:33508: write: broken pipe"

10.0.0.12:33508 is the connection to the caller at 1.2.3.4.

(credit: https://groups.google.com/forum/#!topic/prometheus-users/iAPfckoLYuU)

---

So if you're using federation for mirroring, it's probably best to remove it altogether, and point your Grafana directly to the Prometheus instance which is being federated. You can create a new Prometheus data source, add HTTP or TLS auth, and you're good to go.