Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
CockroachDB logs " ... write: broken pipe" [HAProxy + gRPC] #16084
$ fgrep -F '[config]' node?/logs/cockroach.log
This is a 5 node cluster fronted by HAProxy with config file:
An example of the problem log.
$ fgrep pipe node?/logs/cockroach.log
Some searching revealed similar issues:
Removing the check annotation from the haproxy.cfg file stopped the "broken pipe" error reports.
Verified that HAProxy liveness checks are indeed the cause by adding
to the haproxy.cfg file (after replacing the check annotation).
Error logged by Cockroach becomes:
The "check" annotation is emitted for
cockroach gen haproxy .
Hi @andreimatei .
Cluster is working normally, as far as I can tell. haproxy is distributing requests. (The cluster is running on one machine.)
I find the log messages disconcerting and wanted to know what was causing them.
I guess the point I'm trying to make is that such messages are going to be appearing in people's logs if there are "check" annotations in people's haproxy.cfg files and they might be wondering what was going on.
Oh, so the checks are not failing, as far as HAProxy is concerned. It's just the log spam that's the issue.
@mberhault perhaps the appearance of
Ah, that might explain it.
However, when enabling
And the logs dump a bunch of:
A tcp capture of
So, in summary:
It's annoying that haproxy RSTs all its health check connections instead of shutting them down cleanly (maybe RST is appropriate for the TCP health checks but with
I think that in most cases certain networking errors should be downgraded to