docs: improve documentation and tests for server-side prepared statements #1135
Conversation
Codecov Report
@@ Coverage Diff @@
## master #1135 +/- ##
============================================
+ Coverage 67.25% 67.34% +0.08%
- Complexity 3673 3684 +11
============================================
Files 170 170
Lines 15640 15638 -2
Branches 2531 2531
============================================
+ Hits 10519 10531 +12
+ Misses 3932 3922 -10
+ Partials 1189 1185 -4 |
### Note | ||
> Server side prepared statements are planned only once by the server. This avoids | ||
the cost of replanning the query every time, but also means that the planner | ||
cannot take advantage of the particular parameter values used in a particular |
davecramer
Mar 8, 2018
Member
there are actually a number of plans that are created, not just one. I would have to find the details, but there is some concept of different plans for different values
there are actually a number of plans that are created, not just one. I would have to find the details, but there is some concept of different plans for different values
vlsi
Mar 8, 2018
Author
Member
The implementation is https://github.com/postgres/postgres/blob/9d4649ca49416111aee2c84b7e4441a0b7aa2fac/src/backend/utils/cache/plancache.c#L1034-L1057, however I was unsure if I should modify (update) this comment.
The implementation is https://github.com/postgres/postgres/blob/9d4649ca49416111aee2c84b7e4441a0b7aa2fac/src/backend/utils/cache/plancache.c#L1034-L1057, however I was unsure if I should modify (update) this comment.
davecramer
Mar 8, 2018
Member
Here are the change notes from 9.2:
Prepared statements used to be optimized once, without any knowledge of the parameters' values. With 9.2, the planner will use specific plans regarding to the parameters sent (the query will be planned at execution), except if the query is executed several times and the planner decides that the generic plan is not too much more expensive than the specific plans.
Which conflicts with the comment above.
Here are the change notes from 9.2:
Prepared statements used to be optimized once, without any knowledge of the parameters' values. With 9.2, the planner will use specific plans regarding to the parameters sent (the query will be planned at execution), except if the query is executed several times and the planner decides that the generic plan is not too much more expensive than the specific plans.
Which conflicts with the comment above.
vlsi
Mar 8, 2018
Author
Member
Ok, this summary sounds good.
Ok, this summary sounds good.
Server side prepared statements can improve execution speed as | ||
1. It sends just statement handle (e.g. `S_1`) instead of full SQL text | ||
1. It enables use of binary transfer (e.g. binary int4, binary timestamps, etc), and it is much faster to parse | ||
1. It enables to reuse server-side execution plan |
davecramer
Mar 8, 2018
Member
enables "the" reuse of
enables "the" reuse of
Server side prepared statements can improve execution speed as | ||
1. It sends just statement handle (e.g. `S_1`) instead of full SQL text | ||
1. It enables use of binary transfer (e.g. binary int4, binary timestamps, etc), and it is much faster to parse |
davecramer
Mar 8, 2018
Member
I would say replace ", and it..." with "; the parameters and results are much faster to parse"
I would say replace ", and it..." with "; the parameters and results are much faster to parse"
PreparedStatement ps = con.prepareStatement("select id from rooms where ..."); | ||
if (param instanceof String) { | ||
ps.setString(1, param); | ||
} else if (param instanceof String) { |
bokken
Mar 10, 2018
Member
I think you want instanceof Integer here.
I think you want instanceof Integer here.
The additional doc is great. |
|
||
Even though reusing of the same `PreparedStatement` object good for performance reasons, the driver | ||
is able to server-prepare statements automatically across `connection.prepareStatement(...)` calls. | ||
|
davecramer
Mar 11, 2018
Member
Should be "Even though reuse of the same PreparedStatement object is good"
That being said I'm still unsure what this statement is trying to convey ?
Should be "Even though reuse of the same PreparedStatement object is good"
That being said I'm still unsure what this statement is trying to convey ?
vlsi
Mar 11, 2018
•
Author
Member
The idea is to highlight the difference between
PreparedStatement st = con.prepare("...");
ps.executeQuery();
..
// later in code
ps.executeQuery();
...
// later in code
ps.executeQuery();
vs
PreparedStatement st = con.prepare("...");
ps.executeQuery();
ps.close()
..
// later in code
PreparedStatement st = con.prepare("...");
ps.executeQuery();
ps.close()
...
// later in code
PreparedStatement st = con.prepare("...");
ps.executeQuery();
ps.close()
Certain JDBC drivers require to use exactly the same PreparedStatement
object in order to use server-prepared statements.
The idea is to highlight the difference between
PreparedStatement st = con.prepare("...");
ps.executeQuery();
..
// later in code
ps.executeQuery();
...
// later in code
ps.executeQuery();
vs
PreparedStatement st = con.prepare("...");
ps.executeQuery();
ps.close()
..
// later in code
PreparedStatement st = con.prepare("...");
ps.executeQuery();
ps.close()
...
// later in code
PreparedStatement st = con.prepare("...");
ps.executeQuery();
ps.close()
Certain JDBC drivers require to use exactly the same PreparedStatement
object in order to use server-prepared statements.
davecramer
Mar 11, 2018
Member
Seems this detailed explanation might be appropriate
Seems this detailed explanation might be appropriate
Even though reusing of the same `PreparedStatement` object good for performance reasons, the driver | ||
is able to server-prepare statements automatically across `connection.prepareStatement(...)` calls. | ||
|
||
Server-prepared statements consume memory both at client and server side, so pgjdbc limits the number |
davecramer
Mar 11, 2018
Member
I think "on the client and the server" sounds better
I think "on the client and the server" sounds better
Server-prepared statements consume memory both at client and server side, so pgjdbc limits the number | ||
of server-prepared statements per connection. It can be configured via `preparedStatementCacheQueries` | ||
(default `256`, the number of queries known to pgjdbc), and `preparedStatementCacheSizeMiB` (default `5`, | ||
that is client side cache size in megabytes per connection). Only a subset of `statement cache` is |
davecramer
Mar 11, 2018
Member
that is the client side cache
that is the client side cache
### Deactivation | ||
|
||
There might be cases when you would want to disable use of server-prepared statements. | ||
For instance, if you route connections through a balancer that is unable to speak server-prepared statements, |
davecramer
Mar 11, 2018
Member
"balancer that is incompatible with"
"balancer that is incompatible with"
### Corner cases | ||
|
||
#### DLL | ||
|
davecramer
Mar 11, 2018
Member
DDL ?
DDL ?
SELECT * FROM mytable; -- Does mytable mean app_v1.mytable or app_v2.mytable here? | ||
|
||
Server side prepared statements are linked to database object IDs, so it could fetch data from "old" | ||
`app_v1.mytable` table. It is hard to tell which behavior is expected, however pgjdbc tries to track |
davecramer
Mar 11, 2018
Member
I personally prefer UK spelling of "behaviour"
I personally prefer UK spelling of "behaviour"
|
||
#### Re-execution of failed statements | ||
|
||
It is a pity that a single `cached plan must not change result type` could fail the whole transaction. |
davecramer
Mar 11, 2018
Member
"could cause the whole transaction to fail"
"could cause the whole transaction to fail"
It is a pity that a single `cached plan must not change result type` could fail the whole transaction. | ||
The driver could re-execute the statement automatically in certain cases. | ||
|
||
1. In case the transaction is not failed (e.g. the transaction did not exist before execution of |
davecramer
Mar 11, 2018
Member
is not s/b has not
is not s/b has not
|
||
1. In case the transaction is not failed (e.g. the transaction did not exist before execution of | ||
the statement that caused `cached plan...` error), then pgjdbc re-executes the statement automatically. | ||
This makes application happy, and avoids unnecessary errors. |
davecramer
Mar 11, 2018
Member
This makes the application
This makes the application
#### Replication connection | ||
|
||
PostgreSQL replication connection does not allow to use server side prepared statements, so pgjdbc | ||
uses simple queries in case `replication` connection property is activated. |
davecramer
Mar 11, 2018
Member
in the case where replications
in the case where replications
however the driver has an option to cache simple statements as well. | ||
|
||
You can do that by setting `preferQueryMode` to `extendedCacheEverything`. | ||
Note: the option is more of a diagnostinc/debugging sort, so be careful as you change it. |
davecramer
Mar 11, 2018
Member
so be careful how you use it ?
so be careful how you use it ?
This gets especially painful for batch operations as you don't want to interrupt the batch | ||
by using alternating datatypes. | ||
|
||
The most typical case is as follows (don't **ever** use that in production): |
davecramer
Mar 11, 2018
Member
don't ever use this in production
don't ever use this in production
There are explicit commands to deallocate all server side prepared statements. It would result in | ||
the following server-side error message: `prepared statement name is invalid`. | ||
Of course it could defeat pgjdbc, however there are cases when you need to discard statements (e.g. after lots of DLLs) | ||
|
davecramer
Mar 11, 2018
Member
Sorry one more DLL
Sorry one more DLL
+1 |
Just in case: I have added
with x as (insert into rollbacktest(a, str) values(43, 'abc') returning ${cols}) select * from x
test case, and it does work "as expected".That is it might produce
cached plan...
error in case it is usingreturning *
. When that happens, pgjdbc could auto re-execute the statement (e.g. if no transaction was present) and it does overcome the error automatically.The PR depends on #1137