Timeout on filter requests on large topics

[flamestro]

Version: 2.4.x
Tested Browser: Google Chrome Version 123.0.6312.105 (Official Build) (64-bit)

We discovered something after updating to 2.4.x (any version labeled with 2.4).
When we try to do filter queries on a topic that is large they suddenly stop to work after the update.

After some investigation this seems to be an issue due to the removal of websockets in favor of a standard HTTP request. The http requests timeout after 1 second while the websockets where streaming the data which caused the connection to stay alive. The timeout comes from the backend which closes the connection after one minute.

This can easily be reproduced by creating a large topic with values like {"active": false} in which one value has a true on the active field.

If we now filter over the complete topic (offset earliest) and ask for active true, there will be a timeout.

I think the related commit / series of commits is: dc8d1d1

[weeco]

Hey @flamestro ,
we use HTTP Streaming instead of Websockets for better compatibility in most environments. I believe the timeout may be caused by something higher up in your stack (a cloud LoadBalancer or NGINX). You should receive regular heartbeats via the http stream, and thus not run into idle timeouts. We definitely don't have timeouts for 1s in place in Console.

[flamestro]

I am sorry 1 second was wrong. I Meant 1 minute, 60 seconds.

I actually doubt that this is due to the stack around it as it only happens on the switch from 2.3.x to 2.4.x. I will still try to reproduce this again in a docker compose setup locally with nothing around the console and redpanda backend.

[weeco]

@flamestro We switched from WEbsockets to HTTP streaming with v2.3.x to v2.4.x - so different timeouts may apply. We are using this build in Redpanda Cloud and are able to stream messages with search requests running >1min. Please make sure to use one of the latest v2.4 releases. I think with v2.4.6 we added a heartbeat for specific requests which may help in your situation already - as depending on the search request you would otherwise run into an idle timeout from a TCP loadbalancer.

For other search requests (start offset oldest) we were already sending heartbeats in the form of a progress message. So in case your 60s interrupt is an idle timeout the most recent v2.4.10 release should definitely already help. I'm not entirely sure what v2.4 release you tried.

[flamestro]

Hey @weeco,
I tried every version available with 2.4.x, including 2.4.6. Sadly none of those versions work out for us.

We have quite some entries which might not help with this situation:

This is the loading indicator which takes for ever and then crashes. The same does not happen with 2.3.10

The search filter is only key === "someId".