"My benchmark doesn't show a difference."

"My benchmark doesn't show a difference", is a phrase we have heard before from users looking to switch from an existing pool to HikariCP.

While HikariCP is probably best known for being fast, we have actually spent more effort on achieving that speed within the constraints of providing the highest reliability possible. We are confident that HikariCP is not slightly more reliable, but substantially more reliable than currently available pools.

One of the primary reasons users can't measure a difference between HikariCP and other pools is that most available pools default to a mode of operation where performance is prioritized over reliability.

In contrast, HikariCP has no "unsafe" operational modes -- no way to disable "correct" behavior. As soon as you start turning on the reliability options in other pools to match that of HikariCP, the performance of those pools starts to drop dramatically.

The benchmarks cited on our page are probably overly generous to other pools. In order to measure only the overhead imposed by each pool, the benchmarks are run against a JDBC stub-driver in which every operation is an empty method. When a real driver and DB are put into the loop instead, the difference in results begs believability. But believe them.

So, what is unfair about the typical comparison?

I'm going to talk about HikariCP, Apache DBCP2, and Tomcat DBCP here; talking some about speed, and bringing in the reliability pieces. Here we have run HikariCP-benchmark with the three pools against a real database (MySQL) instead of a stub.

Keep in mind no query is being run here, only getConnection() followed by close().

First, all three pools in default configuration (+ autocommit=false):

Benchmark                       (pool)   Mode   Score
ConnectionBench.cycleCnnection  hikari  thrpt   45289.116  ops/ms
ConnectionBench.cycleCnnection  tomcat  thrpt    2329.692  ops/ms
ConnectionBench.cycleCnnection  dbcp2   thrpt      21.750  ops/ms

DBCP2

What is not visible here is that DBCP2 is generating ~3MB/sec of traffic to the DB, because rollbackOnReturn defaults to true and it is unconditionally rolling back. It does get bonus points for defaulting on the side of safety. Unfortunately, it is not validating connections on borrow.

HikariCP

HikariCP is generating zero traffic to the DB. HikariCP also defaults to "rollback on return" (it can't be turned off because that is the correct behavior for a pool), but it additionally tracks transaction state and does not rollback if the SQL has already been committed or no SQL was run. HikariCP also defaults to "test on borrow" (it can't be turned off...), but employs an optimization that says, "If a connection had activity within the past 1000ms, bypass connection validation."

Tomcat

Tomcat DBCP is also generating zero traffic to the DB, but for a different reason. It simply is not validating connections at all, nor is it rolling back on return.

---

Now, let's try to level the playing field as a little. For Tomcat and DBCP, we need to enable connection validation.

Benchmark                       (pool)   Mode   Score
ConnectionBench.cycleCnnection  hikari  thrpt   45289.116  ops/ms
ConnectionBench.cycleCnnection  tomcat  thrpt    2133.992  ops/ms
ConnectionBench.cycleCnnection  dbcp2   thrpt       5.296  ops/ms

DBCP2

DBCP2 took a big hit here, because it does not have a validation optization like HikariCP. It is still generating ~3MB/sec of traffic to the DB.

At ~5 ops/ms, TPS is capped at a maximum of only 5000 TPS. And 3MB/sec of overhead is generated at that level.

Tomcat

Tomcat DBCP does support a similar optimization to HikariCP, the config goes something like this:

setTestOnBorrow(true);
setValidationInterval(1000);
setValidator( new Validator() {
   public boolean validate(Connection connection, int mode) {
      return connection.isValid(0);
   }
} );

The result is the above performance of 2133.992 ops/ms. Still very good, and likely difficult to measure in most benchmarks.

But we forgot "rollback on return" for Tomcat. Turning that on, performance drops dramatically:

ConnectionBench.cycleCnnection  hikari  thrpt   45289.116  ops/ms
ConnectionBench.cycleCnnection  tomcat  thrpt      20.706  ops/ms
ConnectionBench.cycleCnnection  dbcp2   thrpt       5.296  ops/ms

Tomcat too is now generating ~3MB/sec of traffic to the DB. It does not track transaction state and therefore must unconditionally rollback. I thought maybe enabling "ConnectionState tracking" might help, but it does not.

There is a lot more that HikariCP is doing by default, not covered here ... each one chipping away further at the performance of Tomcat DBCP and Apache DBCP2. These include: guarding against network partitions, checking SQLExceptions for vendor disconnect codes, resetting auto-commit, transaction isolation, catalog, network timeout, tracking open Statements (and closing them), etc.

work in progress

✅ Default, ⭕ Supported, ❌ Not Supported

	HikariCP	Tomcat DBCP	DBCP2
Response time Guarantee	✅	❌	❌
Reset Catalog	✅	⭕	❌
Reset Read-only	✅	⭕	❌
Reset Auto-commit	✅	⭕	❌
Reset Transaction Isolation	✅	⭕	❌
Reset Network Timeout	✅	❌	❌
Rollback-on-return	✅	⭕	✅
Disposable Proxies	✅	⭕	❌
Track/Close Open Statements	✅	❌	⭕
SQLException State Scanning	✅	❌	⭕
Clear SQL warnings	✅	❌	❌

Response time Guarantee

Reset Connection State

ConnectionState* interceptor unconditionally gets/sets state upon every return to the pool and connection creation.

Reset Network Timeout

Rollback-on-return

Disposable Proxies

"Disposable Proxies" prevent code from erroneously using a Connection after returning to the pool. Without them threads can corrupt unrelated transactions.

Track/Close Open Statements

SQLException State Scanning

Check SQLExceptions for SQL-92 and vendor-specific disconnection codes.
DBCP2 comes with no defaults, so it is up to the user to specify all disconnection codes.

Clear SQL warnings