Connection prematurely closed BEFORE response on half closed connection

[asasas234]

Recently, there have been several exceptions about this issue happening every day with my service.

Regarding what this httpclient does, it is like uploading audio files to object storage. And in terms of concurrency, its proportion is very high compared to the entire service.

Currently, my settings are all default settings, but I found the following paragraph in the document:

When you expect a high load, be cautious with a connection pool with a very high value for maximum connections. You might experience reactor.netty.http.client.PrematureCloseException exception with a root cause "Connect Timeout" due to too many concurrent connections opened/acquired.

What I don't understand is why we can't set the number of connections too high? Wouldn't setting it too low actually accelerate this problem?

Do you have any configuration suggestions regarding this issue? What I can confirm is that it's not the remote server closing the connection, but rather a timeout occurring when obtaining the connection on the client side.

[pderop]

I interpret the documentation like this:

if the connection pool size is very high, then if at some point if the connection pool needs to establish a lot of connections simulteneously, then the server (or the proxy) may not be able to accept the connections timely, and in this case some http client requests may fail with a ConnectException: Operation timed out for example, or if https is used, with a SslHandshakeTimeoutException.
Now, if the server is configured with a limit regarding max number of accepted connections, then the server (or the proxy ?) may also possibly immediately close a just accepted connection when the limit is reached, and in this case the client would get a "PrematureCloseException: Connection prematurely closed BEFORE response".

Now, what are the versions of the components you are using (especially the reactor-netty version) ?

Can you share the exact stacktrace ? is it a PrematureCloseException, a Connect exception, a PoolAcquirePendingLimitException ?

thanks

[asasas234]

reactor-netty:1.1.12
reactor-core: 3.5.11
Currently, I have set up a separate HttpClient for the problematic HTTP services. I have also configured their max idle time and LIFO (Last In, First Out) settings. However, I am currently unable to handle new exceptions temporarily.

Thank you for your explanation, I understand that we need to consider not only the max idle time but also the maximum number of connections acceptable by the remote server.

[pderop]

if the exception you got was really a Connection prematurely closed BEFORE response, please have a deep look at https://projectreactor.io/docs/netty/release/reference/index.html#faq.connection-closed

Especially, if you now seem to not have the issue anymore after setting a max idle time in the connection pool, please check #764 (comment):

You can configure a max idle time that is lower than the remote network appliances lowest max idle time. Here's some common ones we've seen:

If many cases, remote servers (or proxy) do have some kind of idle timers, so the key thing is to configure in your connection pool an idle timer that is lower than the remote server idle timer, else there may be a race condition: at some point the HttpClient may reuse a connection, but at the same time, the remote server closes it because it's idle. In this case, you might experience a PrematureCloseException. Setting in the http client pool an idle timer that is lower than the remote server will help a lot in order to avoid that problem.

But please, check the whole FAQ from https://projectreactor.io/docs/netty/release/reference/index.html#faq.connection-closed, because there may be other reasons causing PrematureCloseExceptions.

let me know.
thanks.

[asasas234]

@pderop Currently, I have modified the maximum idle time, and this issue has been resolved.

I have carefully read the introduction in the document about how to locate this issue before. But I can't keep tcpdump or wiretap logs running 24/7. It consumes too much server performance, and this issue only occurs a few times a day. Even if I were to enable it when the problem happens, it would be of no use as the issue has already occurred. So I am not quite accepting the troubleshooting solution mentioned in the documentation.

[pderop]

so, if the issue is resolved, I'm closing this one, we can reopen it if needed.
thanks.

[asasas234]

@pderop This exception has occurred again, it seems that it has not been completely resolved. I have added monitoring to the httpclient connection pool and found that reactor_netty_connection_provider_pending_connections is always 0. Does this mean that the problem is not on the client side but rather that the other end closed the connection? Because this problem occurs infrequently, I cannot keep tcpdump open all the time. Is there any other way to locate this issue?