Inefficient TCP connections use by memberlist transport

[pracucci]

We use a custom transport for memberlist, based on TCP protocol. The main reason why we use TCP is being able to transfer messages which are bigger than the maximum payload of an UDP packet (typically, slightly less than 64KB).

Currently, the TCP transport is implemented in an inefficient way with regards to TCP connection establishment. For every single packet a node needs to transfer to another node, the implementations creates a new TCP connection, writes the packet and then close the connection. See:

dskit/kv/memberlist/tcp_transport.go

Line 438 in ead3f93

func (t *TCPTransport) writeTo(b []byte, addr string) error {

We should consider alternatives like:

• Pros/cons of keeping long-lived TCP connections between nodes, and multiplexing multiple packets over the same connection
• Using a mix of UDP and TCP, selecting the protocol based on the message size (in this case, TLS support wouldn't be available)

[pstibrany]

One alternative which we could explore is reusing gRPC connection and implement Packet (and perhaps Stream, if possible) operations on top of gRPC (as grpc methods). This would give us connection pooling, it would remove the need to configure another port, and it would reuse gRPC TLS settings.

[pstibrany]

My reason to implement tcp transport the way it is, was to keep it simple, with no state on the connection. I agree it's not efficient, and it is time to revisit that decision.

[stevesg]

I recently discovered another issue with this, though I'm unsure whether it's an immediate cause for concern - conntrack table utilization. These short lived connections live on in conntrack for some number of minutes.

A survey of a single node in one of our dev environments at Grafana showed that two thirds (~6000 of ~9000) of the conntrack table were TIME_WAIT dport=7946.

[seizethedave]

Awesome. I would imagine persistent TCP conns would help quite a bit. UDP seems less desirable with intermittently flaky cloud networking, inability to do TLS, ...