New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Moved auto-commit logic from Jaybird to the Firebird. #1

Closed
wants to merge 1 commit into
from

Conversation

Projects
None yet
2 participants
@Smyatkin-Maxim

Jaybird shows horrible performance in auto-commit mode while executing
lots of small queries.
Reasons:

  1. Every time a statement is being executed a new transaction starts and
    commits.
  2. Even if there were no changes at all (e.g., simple select statement)
    Jaybird starts and commits transaction.

Solution: Firebird API has its own auto-commit flag. Firebird knows for
sure if anything did actually change, so Jaybird might rely on it. As a
result:

  1. It saves start and commit calls;
  2. It executes commits only if something really changed;
  3. Internally it works faster even if changes appeared because it doesn't
    restart a transaction.
    Practical results: it allowed me to make OpenCMS work 3-5 times faster
    with Firebird. I suppose it's not the only case, because some systems
    (especially the ones using ORMs) tend to execute lots of small selects.

Main changes in code are:

  1. Removed AutocommitTransactionCoordinator. LocalTransactionCoordinator
    with enabled autoCommit flag does exactly what we need.
  2. When we change autoCoommit to true/false - we commit previous
    transaction manually, so that it restarts with new TPB.
Maxim Smyatkin
Moved auto-commit logic from Jaybird to the Firebird.
Jaybird shows horrible performance in auto-commit mode while executing
lots of small queries.
Reasons:
1. Every time a statement is being executed a new transaction starts and
commits.
2. Even if there were no changes at all (e.g., simple select statement)
Jaybird starts and commits transaction.

Solution: Firebird API has its own auto-commit flag. Firebird knows for
sure if anything did actually change, so Jaybird might rely on it. As a
result:
1. It saves start and commit calls;
2. It executes commits only if something really changed;
3. Internally it works faster even if changes appeared because it doesn't
restart a transaction.
Practical results: it allowed me to make OpenCMS work 3-5 times faster
with Firebird. I suppose it's not the only case, because some systems
(especially the ones using ORMs) tend to execute lots of small selects.

Main changes in code are:
1. Removed AutocommitTransactionCoordinator. LocalTransactionCoordinator
with enabled autoCommit flag does exactly what we need.
2. When we change autoCoommit to true/false - we commit previous
transaction manually, so that it restarts with new TPB.

Smyatkin-Maxim pushed a commit to Smyatkin-Maxim/opencms-core that referenced this pull request May 29, 2015

Maxim Smyatkin
Added non-JPA Firebird support.
The reason is that JPA Firebird driver was very slow due to lots of select
statements per page view being followed by rollbacks (even if it changed nothing).
Non-JPA implementation + auto-commit-friendly Jaybird driver (implemented
here FirebirdSQL/jaybird#1) made page views 3-5
times faster with Firebird.
* If <code>true</code>, this TPB will be set to auto_commit,
* otherwise it won't
*/
public void setAutoCommit(boolean autoCommit) {

This comment has been minimized.

@mrotteveel

mrotteveel May 30, 2015

Member

Given bug JDBC-386: Changes to transaction configuration of a connection are propagated to all connections to the same database this might have unintended side-effects on the transaction config of other connections.

@mrotteveel

mrotteveel May 30, 2015

Member

Given bug JDBC-386: Changes to transaction configuration of a connection are propagated to all connections to the same database this might have unintended side-effects on the transaction config of other connections.

/* (non-Javadoc)
* @see org.firebirdsql.jdbc.InternalTransactionCoordinator#commit()
*/
public void commit() throws SQLException {

This comment has been minimized.

@mrotteveel

mrotteveel May 30, 2015

Member

Loss of existing functionality/requirement of JDBC

@mrotteveel

mrotteveel May 30, 2015

Member

Loss of existing functionality/requirement of JDBC

/* (non-Javadoc)
* @see org.firebirdsql.jdbc.InternalTransactionCoordinator#rollback()
*/
public void rollback() throws SQLException {

This comment has been minimized.

@mrotteveel

mrotteveel May 30, 2015

Member

Loss of existing functionality/requirement of JDBC

@mrotteveel

mrotteveel May 30, 2015

Member

Loss of existing functionality/requirement of JDBC

@mrotteveel

This comment has been minimized.

Show comment
Hide comment
@mrotteveel

mrotteveel May 30, 2015

Member

Created mirror ticket JDBC-399.

Member

mrotteveel commented May 30, 2015

Created mirror ticket JDBC-399.

@mrotteveel mrotteveel self-assigned this May 30, 2015

@mrotteveel

This comment has been minimized.

Show comment
Hide comment
@mrotteveel

mrotteveel Jun 7, 2015

Member

I will start experimenting with this next week. As I commented on the firebird-devel mailinglist, I expect some caveats both in the behavior of Firebird with isc_tpb_autocommit and with the requirements of JDBC, but overall I expect to get this into Jaybird 2.2.9 (possibly as a feature that needs to be enabled by a connection property).

Member

mrotteveel commented Jun 7, 2015

I will start experimenting with this next week. As I commented on the firebird-devel mailinglist, I expect some caveats both in the behavior of Firebird with isc_tpb_autocommit and with the requirements of JDBC, but overall I expect to get this into Jaybird 2.2.9 (possibly as a feature that needs to be enabled by a connection property).

@Smyatkin-Maxim

This comment has been minimized.

Show comment
Hide comment
@Smyatkin-Maxim

Smyatkin-Maxim Jun 7, 2015

Great, thanks!

On Sun, Jun 7, 2015 at 7:26 PM, Mark Rotteveel notifications@github.com
wrote:

I will start experimenting with this next week. As I commented on the
firebird-devel mailinglist, I expect some caveats both in the behavior of
Firebird with isc_tpb_autocommit and with the requirements of JDBC, but
overall I expect to get this into Jaybird 2.2.9 (possibly as a feature that
needs to be enabled by a connection property).


Reply to this email directly or view it on GitHub
#1 (comment).

Thank you!
Smyatkin Maxim, Red Soft Corporation.
https://www.linkedin.com/in/smyatkin

Great, thanks!

On Sun, Jun 7, 2015 at 7:26 PM, Mark Rotteveel notifications@github.com
wrote:

I will start experimenting with this next week. As I commented on the
firebird-devel mailinglist, I expect some caveats both in the behavior of
Firebird with isc_tpb_autocommit and with the requirements of JDBC, but
overall I expect to get this into Jaybird 2.2.9 (possibly as a feature that
needs to be enabled by a connection property).


Reply to this email directly or view it on GitHub
#1 (comment).

Thank you!
Smyatkin Maxim, Red Soft Corporation.
https://www.linkedin.com/in/smyatkin

@mrotteveel

This comment has been minimized.

Show comment
Hide comment
@mrotteveel

mrotteveel Jul 19, 2015

Member

I implemented this as part of http://tracker.firebirdsql.org/browse/JDBC-399 (both in Branch_2_2 and master). I did not use your pull request directly (although I did reference it to see what you changed). Thanks for the suggestion and the ground work!

This option is enabled by specifying the connection property useFirebirdAutocommit=true

With this option, Jaybird will configure the transaction to use isc_tpb_autocommit with autoCommit=true and not commit itself until connection close (or switching to autoCommit=false). The exception is if the statement was of type isc_info_sql_stmt_ddl, in that case Jaybird will commit on statement success and rollback on statement failure (just like it does for all statements in normal auto commit mode). The reason is that Firebird for some DDL commands only executes at a real commit boundary and relying on the Firebird autocommit is insufficient.

On statement completion (as specified in JDBC), result sets will still close unless they are holdable over commit. The result set is only closed clientside, which means that the cursor remains open server side to prevent roundtrips. This may lead to additional resource usage server side unless explicitly closed in the code. Note that any open blobs will be closed client and serverside (until this is improved with JDBC-401).

A connection can be interrogated using FirebirdConnection.isUseFirebirdAutocommit() if it uses isc_tpb_autocommit.

If you manually add isc_tpb_autocommit to the transaction parameter buffer and you enable this option, the isc_tpb_autocommit will be removed from the TPB if autoCommit=false.

Support for this option is experimental, and should only be enabled if you 1) know what you're doing, and 2) if you really need this feature. Internally isc_tpb_autocommit uses commit_retaining, which means that using this feature may increase the transaction gap with associated sweep and garbage collection impact.

Whether or not this improves performance has yet to be tested. The Jaybird testsuite does not show a noticeable difference either way, but most tests only use a a few statements on a single connection.

I will release a snapshot somewhere in the next week, or you can compile it yourself.

Member

mrotteveel commented Jul 19, 2015

I implemented this as part of http://tracker.firebirdsql.org/browse/JDBC-399 (both in Branch_2_2 and master). I did not use your pull request directly (although I did reference it to see what you changed). Thanks for the suggestion and the ground work!

This option is enabled by specifying the connection property useFirebirdAutocommit=true

With this option, Jaybird will configure the transaction to use isc_tpb_autocommit with autoCommit=true and not commit itself until connection close (or switching to autoCommit=false). The exception is if the statement was of type isc_info_sql_stmt_ddl, in that case Jaybird will commit on statement success and rollback on statement failure (just like it does for all statements in normal auto commit mode). The reason is that Firebird for some DDL commands only executes at a real commit boundary and relying on the Firebird autocommit is insufficient.

On statement completion (as specified in JDBC), result sets will still close unless they are holdable over commit. The result set is only closed clientside, which means that the cursor remains open server side to prevent roundtrips. This may lead to additional resource usage server side unless explicitly closed in the code. Note that any open blobs will be closed client and serverside (until this is improved with JDBC-401).

A connection can be interrogated using FirebirdConnection.isUseFirebirdAutocommit() if it uses isc_tpb_autocommit.

If you manually add isc_tpb_autocommit to the transaction parameter buffer and you enable this option, the isc_tpb_autocommit will be removed from the TPB if autoCommit=false.

Support for this option is experimental, and should only be enabled if you 1) know what you're doing, and 2) if you really need this feature. Internally isc_tpb_autocommit uses commit_retaining, which means that using this feature may increase the transaction gap with associated sweep and garbage collection impact.

Whether or not this improves performance has yet to be tested. The Jaybird testsuite does not show a noticeable difference either way, but most tests only use a a few statements on a single connection.

I will release a snapshot somewhere in the next week, or you can compile it yourself.

@mrotteveel mrotteveel closed this Jul 19, 2015

@mrotteveel

This comment has been minimized.

Show comment
Hide comment
Member

mrotteveel commented Jul 24, 2015

@Smyatkin-Maxim

This comment has been minimized.

Show comment
Hide comment
@Smyatkin-Maxim

Smyatkin-Maxim Jul 24, 2015

Okay, great. I'll probably check this out in few weeks, thanks.

Okay, great. I'll probably check this out in few weeks, thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment