Problem with timeouts

[arielzach]

Hello, we are experiencing serious timeout issues. On certain occasions, which we have not been able to identify yet, the timeout is not being honored. We do not observe any CPU problems on the instances to justify these times. For example, in one of our applications, we have set the timeout to 30ms, but we have seen times as long as 600ms. I tried configuring ConnWriteTimeout, but it seems to have no effect, even when set to 1ms. Do you have any idea what the problem could be or what other configuration we could try?

[rueian]

Hi @arielzach, did you set timeout with context.WithTimeout?

[arielzach]

Hello, yes, we are using context.WithTimeout.

[rueian]

The timeout is honored by checking the context you passed in:

rueidis/pipe.go

Line 843 in 2f2f4da

case <-ctxCh:

It is possible to have delays if your CPUs are busy on other things.

[arielzach]

What is the purpose of the ConnWriteTimeout config? Is the timeout respected if there are issues with the connections?

[rueian]

Yes, ConnWriteTimeout is still used to detect hanging connections. It is used in two ways:

1. When pipelining is not started, ConnWriteTimeout is used in net.Conn.SetWriteDeadline which allows us to abort the hanging connection.
2. When pipelining is started, we can't use net.Conn.SetWriteDeadline anymore, so we use periodic PING instead. If the periodic PING does not respond within ConnWriteTimeout then we abort the connection.

[arielzach]

Ok, if I wanted to make sure that the timeout is always met, should I configure ConnWriteTimeout with the same timeout that I use in the context? That way it would also apply to the periodic PING.

[rueian]

The timeout of periodic PING is also checked by watching the context. If your CPUs are really busy, the checking can still be delayed.

In general, I don't recommend setting ConnWriteTimeout to such a small value, that will easily cause all your operations to fail or falsely fail.

BTW, do you use blocking commands such as BLPOP?

[arielzach]

We do not use blocking commands, what other configuration (besides the context) would help us enforce timeouts?

[rueian]

Besides the content deadline, ConnWriteTimeout and Dialer.KeepAlive will affect timeouts. The former works as previously mentioned, and the latter works as the interval of TCP keepalive and the intervals of periodic PINGs.

[arielzach]

In your opinion, with what values should ConnWriteTimeout and Dialer.KeepAlive be configured?

[rueian]

Hi @arielzach, I would recommend keep them default.