New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: improve documentation and tests for server-side prepared statements #1135

Merged
merged 11 commits into from Mar 11, 2018

Conversation

Projects
None yet
4 participants
@vlsi
Member

vlsi commented Mar 8, 2018

Just in case: I have added with x as (insert into rollbacktest(a, str) values(43, 'abc') returning ${cols}) select * from x test case, and it does work "as expected".

That is it might produce cached plan... error in case it is using returning *. When that happens, pgjdbc could auto re-execute the statement (e.g. if no transaction was present) and it does overcome the error automatically.

The PR depends on #1137

@vlsi vlsi added this to the 42.2.2 milestone Mar 8, 2018

@vlsi vlsi requested a review from davecramer Mar 8, 2018

@codecov-io

This comment has been minimized.

codecov-io commented Mar 8, 2018

Codecov Report

Merging #1135 into master will increase coverage by 0.08%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #1135      +/-   ##
============================================
+ Coverage     67.25%   67.34%   +0.08%     
- Complexity     3673     3684      +11     
============================================
  Files           170      170              
  Lines         15640    15638       -2     
  Branches       2531     2531              
============================================
+ Hits          10519    10531      +12     
+ Misses         3932     3922      -10     
+ Partials       1189     1185       -4
### Note
> Server side prepared statements are planned only once by the server. This avoids
the cost of replanning the query every time, but also means that the planner
cannot take advantage of the particular parameter values used in a particular

This comment has been minimized.

@davecramer

davecramer Mar 8, 2018

Member

there are actually a number of plans that are created, not just one. I would have to find the details, but there is some concept of different plans for different values

This comment has been minimized.

@vlsi

This comment has been minimized.

@davecramer

davecramer Mar 8, 2018

Member

Here are the change notes from 9.2:

Prepared statements used to be optimized once, without any knowledge of the parameters' values. With 9.2, the planner will use specific plans regarding to the parameters sent (the query will be planned at execution), except if the query is executed several times and the planner decides that the generic plan is not too much more expensive than the specific plans.

Which conflicts with the comment above.

This comment has been minimized.

@vlsi

vlsi Mar 8, 2018

Member

Ok, this summary sounds good.

Server side prepared statements can improve execution speed as
1. It sends just statement handle (e.g. `S_1`) instead of full SQL text
1. It enables use of binary transfer (e.g. binary int4, binary timestamps, etc), and it is much faster to parse
1. It enables to reuse server-side execution plan

This comment has been minimized.

@davecramer

davecramer Mar 8, 2018

Member

enables "the" reuse of

Server side prepared statements can improve execution speed as
1. It sends just statement handle (e.g. `S_1`) instead of full SQL text
1. It enables use of binary transfer (e.g. binary int4, binary timestamps, etc), and it is much faster to parse

This comment has been minimized.

@davecramer

davecramer Mar 8, 2018

Member

I would say replace ", and it..." with "; the parameters and results are much faster to parse"

PreparedStatement ps = con.prepareStatement("select id from rooms where ...");
if (param instanceof String) {
ps.setString(1, param);
} else if (param instanceof String) {

This comment has been minimized.

@bokken

bokken Mar 10, 2018

Member

I think you want instanceof Integer here.

@bokken

This comment has been minimized.

Member

bokken commented Mar 10, 2018

The additional doc is great.

vlsi added some commits Mar 8, 2018

Even though reusing of the same `PreparedStatement` object good for performance reasons, the driver
is able to server-prepare statements automatically across `connection.prepareStatement(...)` calls.

This comment has been minimized.

@davecramer

davecramer Mar 11, 2018

Member

Should be "Even though reuse of the same PreparedStatement object is good"
That being said I'm still unsure what this statement is trying to convey ?

This comment has been minimized.

@vlsi

vlsi Mar 11, 2018

Member

The idea is to highlight the difference between

PreparedStatement st = con.prepare("...");
ps.executeQuery();
..
// later in code
ps.executeQuery();
...

// later in code
ps.executeQuery();

vs

PreparedStatement st = con.prepare("...");
ps.executeQuery();
ps.close()
..
// later in code
PreparedStatement st = con.prepare("...");
ps.executeQuery();
ps.close()
...

// later in code
PreparedStatement st = con.prepare("...");
ps.executeQuery();
ps.close()

Certain JDBC drivers require to use exactly the same PreparedStatement object in order to use server-prepared statements.

This comment has been minimized.

@davecramer

davecramer Mar 11, 2018

Member

Seems this detailed explanation might be appropriate

Even though reusing of the same `PreparedStatement` object good for performance reasons, the driver
is able to server-prepare statements automatically across `connection.prepareStatement(...)` calls.
Server-prepared statements consume memory both at client and server side, so pgjdbc limits the number

This comment has been minimized.

@davecramer

davecramer Mar 11, 2018

Member

I think "on the client and the server" sounds better

Server-prepared statements consume memory both at client and server side, so pgjdbc limits the number
of server-prepared statements per connection. It can be configured via `preparedStatementCacheQueries`
(default `256`, the number of queries known to pgjdbc), and `preparedStatementCacheSizeMiB` (default `5`,
that is client side cache size in megabytes per connection). Only a subset of `statement cache` is

This comment has been minimized.

@davecramer

davecramer Mar 11, 2018

Member

that is the client side cache

### Deactivation
There might be cases when you would want to disable use of server-prepared statements.
For instance, if you route connections through a balancer that is unable to speak server-prepared statements,

This comment has been minimized.

@davecramer

davecramer Mar 11, 2018

Member

"balancer that is incompatible with"

### Corner cases
#### DLL

This comment has been minimized.

@davecramer
SELECT * FROM mytable; -- Does mytable mean app_v1.mytable or app_v2.mytable here?
Server side prepared statements are linked to database object IDs, so it could fetch data from "old"
`app_v1.mytable` table. It is hard to tell which behavior is expected, however pgjdbc tries to track

This comment has been minimized.

@davecramer

davecramer Mar 11, 2018

Member

I personally prefer UK spelling of "behaviour"

#### Re-execution of failed statements
It is a pity that a single `cached plan must not change result type` could fail the whole transaction.

This comment has been minimized.

@davecramer

davecramer Mar 11, 2018

Member

"could cause the whole transaction to fail"

It is a pity that a single `cached plan must not change result type` could fail the whole transaction.
The driver could re-execute the statement automatically in certain cases.
1. In case the transaction is not failed (e.g. the transaction did not exist before execution of

This comment has been minimized.

@davecramer

davecramer Mar 11, 2018

Member

is not s/b has not

1. In case the transaction is not failed (e.g. the transaction did not exist before execution of
the statement that caused `cached plan...` error), then pgjdbc re-executes the statement automatically.
This makes application happy, and avoids unnecessary errors.

This comment has been minimized.

@davecramer

davecramer Mar 11, 2018

Member

This makes the application

#### Replication connection
PostgreSQL replication connection does not allow to use server side prepared statements, so pgjdbc
uses simple queries in case `replication` connection property is activated.

This comment has been minimized.

@davecramer

davecramer Mar 11, 2018

Member

in the case where replications

however the driver has an option to cache simple statements as well.
You can do that by setting `preferQueryMode` to `extendedCacheEverything`.
Note: the option is more of a diagnostinc/debugging sort, so be careful as you change it.

This comment has been minimized.

@davecramer

davecramer Mar 11, 2018

Member

so be careful how you use it ?

This gets especially painful for batch operations as you don't want to interrupt the batch
by using alternating datatypes.
The most typical case is as follows (don't **ever** use that in production):

This comment has been minimized.

@davecramer

davecramer Mar 11, 2018

Member

don't ever use this in production

There are explicit commands to deallocate all server side prepared statements. It would result in
the following server-side error message: `prepared statement name is invalid`.
Of course it could defeat pgjdbc, however there are cases when you need to discard statements (e.g. after lots of DLLs)

This comment has been minimized.

@davecramer

davecramer Mar 11, 2018

Member

Sorry one more DLL

vlsi added some commits Mar 11, 2018

@davecramer

This comment has been minimized.

Member

davecramer commented Mar 11, 2018

+1

@vlsi vlsi merged commit 4204f09 into pgjdbc:master Mar 11, 2018

1 check was pending

continuous-integration/travis-ci/pr The Travis CI build is in progress
Details

rhavermans added a commit to bolcom/pgjdbc that referenced this pull request Jul 13, 2018

rhavermans added a commit to bolcom/pgjdbc that referenced this pull request Jul 13, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment