Description
Hi!
The Extended Query protocol enables drivers to submit multiple BIND messages before SYNC. One of the big benefits of using Extended Queries is that you can efficiently send a batch of executions without waiting for round trips for each execution. Pairing that with prepared statements and some simplifications: you send a single PARSE, a bunch of BIND/EXECUTE and a SYNC to find out how things went.
In other words, you'd be able to support something like the following without needing 4 entire round trips. (I'm not recommending this API since it would be a terrible breaking change.)
await client.query({
name: 'my_query',
text: 'insert sometable (id, val) values ($1, $2)'
values: [
[ 1, 'asdf' ],
[ 2, 'fdsa' ],
[ 3, 'qwer' ],
[ 4, 'uytr' ]
]
})
For more information, check out how the JDBC Postgres driver handles a batched execution. There are a few layers to dig through, but this appears to be the core of the code that sends a batch of messages and subsequently sends a single SYNC. NOTE: their driver imposes a limit of 128 records per batch as (apparently) further batching does not improve performance.
Activity
rightaway commentedon Sep 8, 2021
Isn't it already possible in this case? #1190 (comment)
mpareja commentedon Sep 8, 2021
While the two approaches look similar, there are very different performance characteristics. Sending a large number of SQL statements with different parameters will perform much worse than sending a single prepared statement and binding many parameters to it.
brianc commentedon Sep 8, 2021
yeah a proper "batched query" would be nice. Probably a separate object you pass to
client.query
orpool.query
that was likeThen the batch query execution could throw if this is false for some validation:
something like that. Then it would be explicit.
damianobarbati commentedon Aug 28, 2022
@brianc is this supported?
brianc commentedon Aug 29, 2022
from a protocol perspective, yes. But I haven't actually implemented the code yet.
iamkhush commentedon Aug 29, 2022
Hi @brianc , I am interesting contributing since I believe this would be helpful for my usecase in production.
Any guidance would be appreciated.
brianc commentedon Aug 29, 2022
That'd be cool! I'd suggest making this a separate module like
pg-cursor
orpg-query-stream
. It's fine to inline it into this repo as another module here, but best to keep it out of core ofpg
to keep bloat to a minimum. So, w/ that in mind we can look atpg-cursor
to see how to do something like this...Basically anything passed to
client.query
will be sniffed to see if it has asubmit
function. If it does, that function is called, passing in theconnection
object. From that point forward it can fully take over the underlying connection object (which is basically low level functions to send/receieve postgres packets directly) and do anything it wants. Once it emitsend
it'll need to clean up after itself. It's not the worlds most well-designed API, particularly from my current skill level, but it is what I came up w/ many years ago and in the interest of backwards compatibility it is what is there today.brianc commentedon Aug 29, 2022
https://github.com/brianc/node-postgres/blob/master/packages/pg-cursor/index.js#L42
This is a good point of reference.
aleclarson commentedon Sep 7, 2022
It would be great if this feature was implemented with support for multiple queries, rather than multiple value arrays only.
charmander commentedon Sep 7, 2022
@aleclarson That would be pipelining, not multiple bind.
abenhamdine commentedon Nov 7, 2022
for pipelining, see the experiment in #2706
iamkhush commentedon Mar 30, 2023
Hey @brianc @mpareja I have put up a PR.
Can you please have a look at the PR and provide feedback
Remarkably, the bench.ts file shows that inserts are getting 100% increase in query per second.