sql: add a sql.defaults.conn_results_buffer_size cluster setting #32366
Conversation
|
This change is |
|
cc @danhhz |
andreimatei
force-pushed
the
conn-buff
branch
3 times, most recently
from
a9dddce
to
b2a3ee2
Compare
|
changefeedccl stuff lgtm. I'll defer to @knz for the rest |
There was a problem hiding this comment.
LGTM modulo improvement of the setting name + description, see my comment below.
However I am not sure a cluster setting is the best way to go.
If it was just me I would have went the route of a client status parameter (a new case in parseOptions(), pgwire/server.go), so that the buffer can be set to a different value per session.
Reviewed 11 of 11 files at r1, 5 of 5 files at r2.
Reviewable status:complete! 0 of 0 LGTMs obtained
pkg/sql/pgwire/conn.go, line 107 at r1 (raw file):
// corresponding session variable. var connResultsBufferSize = settings.RegisterByteSizeSetting( "sql.defaults.conn_results_buffer_size",
There is a tech note / RFC / guideline somewhere that should have informed you that we don't shorten words in setting names. So the setting name should be
sql.defaults.connection_results_buffer.size
(I think that this is even too verbose -- we have just one results buffer so it could be sql.defaults.results_buffer.size
pkg/sql/pgwire/conn.go, line 108 at r1 (raw file):
var connResultsBufferSize = settings.RegisterByteSizeSetting( "sql.defaults.conn_results_buffer_size", "the size of each connection's results buffer. Within a query (or a batch "+
-
no "the" at the beginning
-
I think the text is too long -- you could summarize to:
"size of the buffer that accumulates query results before they are sent to the client. Note that auto-retries by the server only happen on buffered results, so reducing this size exposes clients to more manual retries."
-
don't use
\n(we don't use newlines in setting descriptions) -
if you keep your text use "the server" instead of "CRDB"
pkg/sql/pgwire/conn.go, line 109 at r1 (raw file):
"sql.defaults.conn_results_buffer_size", "the size of each connection's results buffer. Within a query (or a batch "+ "of queries), CRDB buffers results until the buffer overflows, at which point "+
... until the buffer is full, at which point ...
pkg/sql/pgwire/conn.go, line 112 at r1 (raw file):
"the results are delivered to the client. Once any results are delivered, "+ "CRDB can no longer perform automatic retries for the transactions in "+ "question. On the other hand, increasing the buffer size can increase the "+
... increasing the buffer size increases the delay until the client receives the first result row.
pkg/sql/pgwire/conn.go, line 115 at r1 (raw file):
"latency the client perceives until the first result row for a query is "+ "delivered.\n"+ "Updating the setting only affects future connections.\n"+
... only affects new connections.
andreimatei
force-pushed
the
conn-buff
branch
from
b2a3ee2
to
00a227e
Compare
There was a problem hiding this comment.
I agree that having control over buffering per-session would be nice, but I'll let someone else do it. Even when we do, a configurable default still makes sense (which is why I put the "defaults" in the setting name. In fact, ideally the control would be per-txn, but how to do that is less clear.
Reviewable status:
pkg/sql/pgwire/conn.go, line 107 at r1 (raw file):
Previously, knz (kena) wrote…
There is a tech note / RFC / guideline somewhere that should have informed you that we don't shorten words in setting names. So the setting name should be
sql.defaults.connection_results_buffer.size(I think that this is even too verbose -- we have just one results buffer so it could be
sql.defaults.results_buffer.size
sql.defaults.results_buffer.size it is
pkg/sql/pgwire/conn.go, line 108 at r1 (raw file):
Previously, knz (kena) wrote…
no "the" at the beginning
I think the text is too long -- you could summarize to:
"size of the buffer that accumulates query results before they are sent to the client. Note that auto-retries by the server only happen on buffered results, so reducing this size exposes clients to more manual retries."
don't use
\n(we don't use newlines in setting descriptions)if you keep your text use "the server" instead of "CRDB"
rephrased.
pkg/sql/pgwire/conn.go, line 109 at r1 (raw file):
Previously, knz (kena) wrote…
... until the buffer is full, at which point ...
Done.
pkg/sql/pgwire/conn.go, line 112 at r1 (raw file):
Previously, knz (kena) wrote…
... increasing the buffer size increases the delay until the client receives the first result row.
Done.
pkg/sql/pgwire/conn.go, line 115 at r1 (raw file):
Previously, knz (kena) wrote…
... only affects new connections.
Done.
There was a problem hiding this comment.
Reviewed 2 of 7 files at r3, 5 of 5 files at r4.
Reviewable status:
pkg/sql/pgwire/server.go, line 54 at r4 (raw file):
var connResultsBufferSize = settings.RegisterByteSizeSetting( "sql.defaults.conn_results_buffer_size", "the size of each connection's results buffer. Within a query (or a batch "+
you forgot to commit the changes to this file.
... controlling the buffering of results on (all) future connections. This has come up with users enough to warrant it: some would rather buffer more to get more auto-retries. Controlling this for each connection individually is not done. Release note (general change): A new cluster setting (sql.defaults.conn_results_buffer_size) can be used to control server-side buffering of results.
The reference from a connection to an ExecutorConfig is no longer needed. Release note: None
andreimatei
force-pushed
the
conn-buff
branch
from
00a227e
to
f25ac68
Compare
There was a problem hiding this comment.
Reviewable status:
pkg/sql/pgwire/server.go, line 54 at r4 (raw file):
Previously, knz (kena) wrote…
you forgot to commit the changes to this file.
done. damn it, I thought I paid attention to this. thanks.
There was a problem hiding this comment.
Reviewed 6 of 6 files at r5, 5 of 5 files at r6.
Reviewable status:
32366: sql: add a sql.defaults.conn_results_buffer_size cluster setting r=andreimatei a=andreimatei ... controlling the buffering of results on (all) future connections. This has come up with users enough to warrant it: some would rather buffer more to get more auto-retries. Controlling this for each connection individually is not done. Release note: None Co-authored-by: Andrei Matei <andrei@cockroachlabs.com>
Build succeeded |
Recently cockroachdb#32366 added a cluster setting that changes the default pgwire result buffer size for all future connections. A TODO was left to allow controlling it on a connection basis and this commit resolves that TODO. Adding something like "?results_buffer_size=1KiB" to the connection string will set the pgwire result buffer size for that connection, overriding the default specified in the cluster setting. Users may want to set this to a larger value to get more auto-retries or to a smaller value with the (still experimental) core version of changefeeds. Also switch the changefeed unit tests to use this instead of the cluster setting. Release note (sql change): pgwire result buffer size can now be controlled on a per-connection basis with the results_buffer_size connection string param
Recently cockroachdb#32366 added a cluster setting that changes the default pgwire result buffer size for all future connections. A TODO was left to allow controlling it on a connection basis and this commit resolves that TODO. Adding something like "?results_buffer_size=1KiB" to the connection string will set the pgwire result buffer size for that connection, overriding the default specified in the cluster setting. Users may want to set this to a larger value to get more auto-retries or to a smaller value with the (still experimental) core version of changefeeds. Also switch the changefeed unit tests to use this instead of the cluster setting. Release note (sql change): pgwire result buffer size can now be controlled on a per-connection basis with the results_buffer_size connection string param
34385: pgwire: allow per-connection control of result buffer size r=knz a=danhhz Recently #32366 added a cluster setting that changes the default pgwire result buffer size for all future connections. A TODO was left to allow controlling it on a connection basis and this commit resolves that TODO. Adding something like "?results_buffer_size=1KiB" to the connection string will set the pgwire result buffer size for that connection, overriding the default specified in the cluster setting. Users may want to set this to a larger value to get more auto-retries or to a smaller value with the (still experimental) core version of changefeeds. Also switch the changefeed unit tests to use this instead of the cluster setting. Release note (sql change): pgwire result buffer size can now be controlled on a per-connection basis with the results_buffer_size connection string param Co-authored-by: Daniel Harrison <daniel.harrison@gmail.com>
... controlling the buffering of results on (all) future connections.
This has come up with users enough to warrant it: some would rather
buffer more to get more auto-retries.
Controlling this for each connection individually is not done.
Release note: None