-
Notifications
You must be signed in to change notification settings - Fork 2.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Trino-cli has no option to accept targetResultSize query parameter #22303
Comments
What is the use case you are trying to solve with exposing this parameter? The protocol is already conversational so there is no real limit for the overall data returned. |
Hi @mosabua Currently in below mentioned code I see that for query get request to fetch resultset there is a queryparameter targetResultSize defined with a default size as 16MB i.e 16MB of result set would be sent to client from coordinator per get request. Another thing to note here is this parameter is never set i.e it's always null because there is no way this parameter can be set from client. but I see a condition to check take min of this value and max target size which is 128MB if it is set. If this parameter can be set through client the queries that would fetch large resultset from coordinator mininization of the get request calls can be done which would reduce the wait time and queries would complete faster. ExecuteStatementResource.java( trino/core/trino-main/src/main/java/io/trino/server/protocol/ExecutingStatementResource.java Line 216 in 1b0a011
please let me know your comments. |
This might be something we can do in our current work on improving client protocol performance with Project Swift. Note that we don't know if increasing the size does indeed improve performance. Did you test this and can report any findings? Also .. ideally users would not have to configure such things just to get a faster connection. |
What I have observed is in code targetResultSize was set as QueryParam and was never getting used or set from trino client while running a query, so I was just curious on why it was not done. I have run a query which returns approximately 6GB of data with 7.7Million rows on a trino cluster from DBVisualizer using trino-jdbc client driver jar, these are my findigs:
|
@mosabua I can't agree with these numbers - I've been benchmarking changing this value and there is a diminishing improvement for values over 32 MB. The buffering and compression happens on the coordinator so setting this value high, puts an additional pressure on the coordinator. That doesn't scale well. |
My trino cluster has 6 nodes = 5 workers and 1 coordinator(doesn't act as worker) where each node has 1TB memory and 128 CPU where I have run the query. |
In query result GET request from coordinator it has a query parameter targetResultSize which is per get request how much a coordinator can send to client.
I see there is no way to configure it from trino client.
The text was updated successfully, but these errors were encountered: