Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connection is not available error even if there idle connections. #731

Closed
buremba opened this issue Sep 26, 2016 · 12 comments
Closed

Connection is not available error even if there idle connections. #731

buremba opened this issue Sep 26, 2016 · 12 comments

Comments

@buremba
Copy link

buremba commented Sep 26, 2016

I use the default configuration and limit the maximum connection to 30.

When I start the application, it works fine but after some time, I'm getting timeout errors even if there is only one running query.

Normally, Hikari creates 30 connections but eventually it closes them in lifecycle process and keep only one connection in total.

Here is the logs:

2016-09-26T04:09:23.875+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - After adding stats (total=30, active=0, idle=30, waiting=0)
2016-09-26T04:09:23.979+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@3a3991: (connection has passed maxLifetime)
2016-09-26T04:09:24.085+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@69a3a909: (connection has passed maxLifetime)
2016-09-26T04:09:24.239+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@d63650f: (connection has passed maxLifetime)
2016-09-26T04:09:24.241+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@3884f31d: (connection has passed maxLifetime)
2016-09-26T04:09:24.253+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@2bf32fb: (connection has passed maxLifetime)
2016-09-26T04:09:24.449+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@16676f84: (connection has passed maxLifetime)
2016-09-26T04:09:24.454+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@7630a9d8: (connection has passed maxLifetime)
2016-09-26T04:09:24.477+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@847598b: (connection has passed maxLifetime)
2016-09-26T04:09:24.522+0300    DEBUG   generic-jdbc-query-executor housekeeper com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Pool stats (total=17, active=0, idle=17, waiting=0)
2016-09-26T04:09:24.522+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@73b73056: (connection has passed maxLifetime)
2016-09-26T04:09:24.528+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@5d7e4b1e
2016-09-26T04:09:24.535+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@53b9aec1
2016-09-26T04:09:24.540+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@5786206e
2016-09-26T04:09:24.546+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@1284a4fa
2016-09-26T04:09:24.551+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@3d8d3b0f
2016-09-26T04:09:24.555+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@23dc55d9
2016-09-26T04:09:24.560+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@453f949c
2016-09-26T04:09:24.566+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@72ab5bb5
2016-09-26T04:09:24.568+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@75043eb5: (connection has passed maxLifetime)
2016-09-26T04:09:24.571+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@26aa53ac
2016-09-26T04:09:24.576+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@6799a83d
2016-09-26T04:09:24.584+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@5e3b5b26
2016-09-26T04:09:24.590+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@76e5c5a7
2016-09-26T04:09:24.595+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@226cfe4
2016-09-26T04:09:24.595+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - After adding stats (total=28, active=0, idle=28, waiting=0)
2016-09-26T04:09:24.604+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@23dc08a4: (connection has passed maxLifetime)
2016-09-26T04:09:25.571+0300    DEBUG   generic-jdbc-query-executor housekeeper com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Pool stats (total=1, active=1, idle=0, waiting=0)
2016-09-26T04:09:52.629+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@419d3e64: (connection has passed maxLifetime)
2016-09-26T04:09:52.702+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@6ee17780: (connection has passed maxLifetime)
2016-09-26T04:09:52.728+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@7924766: (connection has passed maxLifetime)
2016-09-26T04:09:52.736+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@7d2175c9: (connection has passed maxLifetime)
2016-09-26T04:09:52.769+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@2fcff8b6: (connection has passed maxLifetime)
2016-09-26T04:09:52.831+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@43d88efc: (connection has passed maxLifetime)
2016-09-26T04:09:52.892+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@ebb1aee: (connection has passed maxLifetime)
2016-09-26T04:09:53.063+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@63738bb: (connection has passed maxLifetime)
2016-09-26T04:09:53.132+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@6a216854: (connection has passed maxLifetime)
2016-09-26T04:09:53.158+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@6635de94: (connection has passed maxLifetime)
2016-09-26T04:09:53.229+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@6e6b9017: (connection has passed maxLifetime)
2016-09-26T04:09:53.229+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@75f522e8: (connection has passed maxLifetime)
2016-09-26T04:09:53.356+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@21917991: (connection has passed maxLifetime)
2016-09-26T04:09:53.362+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@6df96001: (connection has passed maxLifetime)
2016-09-26T04:09:53.522+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@1754fc2b: (connection has passed maxLifetime)
2016-09-26T04:09:53.537+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@52182973: (connection has passed maxLifetime)
2016-09-26T04:09:53.684+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@666c6598: (connection has passed maxLifetime)
2016-09-26T04:09:53.706+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@3975fb68: (connection has passed maxLifetime)
2016-09-26T04:09:53.737+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@704dcb78: (connection has passed maxLifetime)
2016-09-26T04:09:53.748+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@10fabe6c: (connection has passed maxLifetime)
2016-09-26T04:09:53.775+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@41be86de: (connection has passed maxLifetime)
2016-09-26T04:09:53.776+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@6493fb67: (connection has passed maxLifetime)
2016-09-26T04:09:53.789+0300    DEBUG   generic-jdbc-query-executor housekeeper com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Pool stats (total=18, active=0, idle=18, waiting=0)
2016-09-26T04:09:53.794+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@4cacea28
2016-09-26T04:09:53.798+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@11dd949b
2016-09-26T04:09:53.802+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@4bcb036a
2016-09-26T04:09:53.810+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@50e9539e
2016-09-26T04:09:53.816+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@1f70b96c
2016-09-26T04:09:53.821+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@6c4238c8
2016-09-26T04:09:53.825+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@7a353e24
2016-09-26T04:09:53.830+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@76376b51
2016-09-26T04:09:53.835+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@9102cb6
2016-09-26T04:09:53.854+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@62d6147f
2016-09-26T04:09:53.856+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@514d611
2016-09-26T04:09:53.865+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@1a7909eb
2016-09-26T04:09:53.866+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - After adding stats (total=30, active=0, idle=30, waiting=0)
2016-09-26T04:09:54.260+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@6b4927d7: (connection has passed maxLifetime)
2016-09-26T04:09:54.413+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@13c8be9c: (connection has passed maxLifetime)
2016-09-26T04:09:54.439+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@60ac73aa: (connection has passed maxLifetime)
2016-09-26T04:09:54.451+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@45f03bd9: (connection has passed maxLifetime)
2016-09-26T04:09:54.526+0300    DEBUG   generic-jdbc-query-executor housekeeper com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Pool stats (total=13, active=0, idle=13, waiting=0)
2016-09-26T04:09:54.533+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@7f8a79bc
2016-09-26T04:09:54.539+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@714d883d
2016-09-26T04:09:54.545+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@757212cc
2016-09-26T04:09:54.551+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@30708206
2016-09-26T04:09:54.556+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@1c1b4a2d
2016-09-26T04:09:54.559+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@18a35f3
2016-09-26T04:09:54.563+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@640ba37c
2016-09-26T04:09:54.567+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@505f7d49
2016-09-26T04:09:54.571+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@16531f4d
2016-09-26T04:09:54.576+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@462695d9
2016-09-26T04:09:54.584+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@7f0fce22
2016-09-26T04:09:54.591+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@541141bf
2016-09-26T04:09:54.598+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@5a368507
2016-09-26T04:09:54.602+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@28cc5bc5
2016-09-26T04:09:54.610+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@29b3fe20
2016-09-26T04:09:54.615+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@77f15e10
2016-09-26T04:09:54.620+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@1901f354
2016-09-26T04:09:54.620+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - After adding stats (total=30, active=0, idle=30, waiting=0)
2016-09-26T04:09:55.575+0300    DEBUG   generic-jdbc-query-executor housekeeper com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Pool stats (total=1, active=1, idle=0, waiting=0)
2016-09-26T04:10:22.343+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@69d4fa0f: (connection has passed maxLifetime)
2016-09-26T04:10:22.486+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@7b721543: (connection has passed maxLifetime)
2016-09-26T04:10:22.495+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@114fedf5: (connection has passed maxLifetime)
2016-09-26T04:10:22.515+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@18b1e1a: (connection has passed maxLifetime)
2016-09-26T04:10:22.549+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@65eb39de: (connection has passed maxLifetime)
2016-09-26T04:10:22.649+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@df2abea: (connection has passed maxLifetime)
2016-09-26T04:10:22.814+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@2592fde1: (connection has passed maxLifetime)
2016-09-26T04:10:22.957+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@33da078f: (connection has passed maxLifetime)
2016-09-26T04:10:23.017+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@7ddfff2d: (connection has passed maxLifetime)
2016-09-26T04:10:23.071+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@29847ed9: (connection has passed maxLifetime)
2016-09-26T04:10:23.093+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@53b9aec1: (connection has passed maxLifetime)
2016-09-26T04:10:23.128+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@6be5d649: (connection has passed maxLifetime)
2016-09-26T04:10:23.176+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@4fcd2621: (connection has passed maxLifetime)
2016-09-26T04:10:23.310+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@5d7e4b1e: (connection has passed maxLifetime)
2016-09-26T04:10:23.334+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@5e3b5b26: (connection has passed maxLifetime)
2016-09-26T04:10:23.400+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@5786206e: (connection has passed maxLifetime)
2016-09-26T04:10:23.497+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@6799a83d: (connection has passed maxLifetime)
2016-09-26T04:10:23.511+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@7a889e28: (connection has passed maxLifetime)
2016-09-26T04:10:23.533+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@1a50d9af: (connection has passed maxLifetime)
2016-09-26T04:10:23.551+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@76e5c5a7: (connection has passed maxLifetime)
2016-09-26T04:10:23.655+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@67d70633: (connection has passed maxLifetime)
2016-09-26T04:10:23.658+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@58e15828: (connection has passed maxLifetime)
2016-09-26T04:10:23.729+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@453f949c: (connection has passed maxLifetime)
2016-09-26T04:10:23.748+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@7c2fbf84: (connection has passed maxLifetime)
2016-09-26T04:10:23.755+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@211d6cf6: (connection has passed maxLifetime)
2016-09-26T04:10:23.765+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@1284a4fa: (connection has passed maxLifetime)
2016-09-26T04:10:23.793+0300    DEBUG   generic-jdbc-query-executor housekeeper com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Pool stats (total=12, active=0, idle=12, waiting=0)
2016-09-26T04:10:23.799+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@23dc55d9: (connection has passed maxLifetime)
2016-09-26T04:10:23.813+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@430d2704
2016-09-26T04:10:23.826+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@4abec178
2016-09-26T04:10:23.833+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@1f152ca1
2016-09-26T04:10:23.836+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@53b998e8
2016-09-26T04:10:23.847+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@6651d141
2016-09-26T04:10:23.854+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@14929c4e
2016-09-26T04:10:23.870+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@afb89ed
2016-09-26T04:10:23.899+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@60d54bdf
2016-09-26T04:10:23.904+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@6b3fa525
2016-09-26T04:10:23.915+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@381d9ec8
2016-09-26T04:10:23.919+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@2a9ab87
2016-09-26T04:10:23.950+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@592c8ab5
2016-09-26T04:10:23.985+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@648185fb
2016-09-26T04:10:24.012+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@54fdf8da
2016-09-26T04:10:24.030+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@226cfe4: (connection has passed maxLifetime)
2016-09-26T04:10:24.048+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@5fc04255
2016-09-26T04:10:24.054+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@3d8d3b0f: (connection has passed maxLifetime)
2016-09-26T04:10:24.089+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@5c400805
2016-09-26T04:10:24.096+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@405839d7
2016-09-26T04:10:24.105+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@5985301a
2016-09-26T04:10:24.105+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - After adding stats (total=30, active=0, idle=30, waiting=0)
2016-09-26T04:10:24.276+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@26aa53ac: (connection has passed maxLifetime)
2016-09-26T04:10:24.283+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@72ab5bb5: (connection has passed maxLifetime)
2016-09-26T04:10:24.529+0300    DEBUG   generic-jdbc-query-executor housekeeper com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Pool stats (total=17, active=0, idle=17, waiting=0)
2016-09-26T04:10:24.541+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@595afb40
2016-09-26T04:10:24.552+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@579d1e1b
2016-09-26T04:10:24.557+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@53a29030
2016-09-26T04:10:24.561+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@16a01420
2016-09-26T04:10:24.565+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@182e6d4f
2016-09-26T04:10:24.569+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@27fcc597
2016-09-26T04:10:24.573+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@6098e248
2016-09-26T04:10:24.577+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@768feb6f
2016-09-26T04:10:24.583+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@1da0a4e4
2016-09-26T04:10:24.588+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@5c311aba
2016-09-26T04:10:24.593+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@7b9c29d6
2016-09-26T04:10:24.598+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@72c17159
2016-09-26T04:10:24.603+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@6ade7c61
2016-09-26T04:10:24.603+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - After adding stats (total=30, active=0, idle=30, waiting=0)
2016-09-26T04:10:25.577+0300    DEBUG   generic-jdbc-query-executor housekeeper com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Pool stats (total=1, active=1, idle=0, waiting=0)
2016-09-26T04:10:52.396+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@6c4238c8: (connection has passed maxLifetime)
2016-09-26T04:10:52.451+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@50e9539e: (connection has passed maxLifetime)
2016-09-26T04:10:52.545+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@4bcb036a: (connection has passed maxLifetime)
2016-09-26T04:10:52.629+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@4cacea28: (connection has passed maxLifetime)
2016-09-26T04:10:52.643+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@1f70b96c: (connection has passed maxLifetime)
2016-09-26T04:10:52.725+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@7a353e24: (connection has passed maxLifetime)
2016-09-26T04:10:52.997+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@11dd949b: (connection has passed maxLifetime)
2016-09-26T04:10:53.081+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@1c1b4a2d: (connection has passed maxLifetime)
2016-09-26T04:10:53.216+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@1a7909eb: (connection has passed maxLifetime)
2016-09-26T04:10:53.223+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@62d6147f: (connection has passed maxLifetime)
2016-09-26T04:10:53.244+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@514d611: (connection has passed maxLifetime)
2016-09-26T04:10:53.334+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@76376b51: (connection has passed maxLifetime)
2016-09-26T04:10:53.473+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@757212cc: (connection has passed maxLifetime)
2016-09-26T04:10:53.506+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@7f8a79bc: (connection has passed maxLifetime)
2016-09-26T04:10:53.564+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@29b3fe20: (connection has passed maxLifetime)
2016-09-26T04:10:53.620+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@30708206: (connection has passed maxLifetime)
2016-09-26T04:10:53.683+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@9102cb6: (connection has passed maxLifetime)
2016-09-26T04:10:53.749+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@18a35f3: (connection has passed maxLifetime)
2016-09-26T04:10:53.792+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@462695d9: (connection has passed maxLifetime)
2016-09-26T04:10:53.794+0300    DEBUG   generic-jdbc-query-executor housekeeper com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Pool stats (total=18, active=0, idle=18, waiting=0)
2016-09-26T04:10:53.800+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@6d6a8686
2016-09-26T04:10:53.804+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@1e09997a
2016-09-26T04:10:53.807+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@76330d21
2016-09-26T04:10:53.812+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@2f32bd4d
2016-09-26T04:10:53.820+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@4814f6b
2016-09-26T04:10:53.827+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@2063d13
2016-09-26T04:10:53.830+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@59dd925e
2016-09-26T04:10:53.833+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@18198fd
2016-09-26T04:10:53.842+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@70368c00
2016-09-26T04:10:53.850+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@77f15e10: (connection has passed maxLifetime)
2016-09-26T04:10:53.851+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@7c98fb4b
2016-09-26T04:10:53.875+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@69c4ec4
2016-09-26T04:10:53.885+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@11c44c6c
2016-09-26T04:10:53.885+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - After adding stats (total=30, active=0, idle=30, waiting=0)
2016-09-26T04:10:53.938+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@640ba37c: (connection has passed maxLifetime)
2016-09-26T04:10:53.947+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@714d883d: (connection has passed maxLifetime)
2016-09-26T04:10:54.218+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@1901f354: (connection has passed maxLifetime)
2016-09-26T04:10:54.280+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@505f7d49: (connection has passed maxLifetime)
2016-09-26T04:10:54.280+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@7f0fce22: (connection has passed maxLifetime)
2016-09-26T04:10:54.340+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@541141bf: (connection has passed maxLifetime)
2016-09-26T04:10:54.438+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@16531f4d: (connection has passed maxLifetime)
2016-09-26T04:10:54.447+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@28cc5bc5: (connection has passed maxLifetime)
2016-09-26T04:10:54.510+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@5a368507: (connection has passed maxLifetime)
2016-09-26T04:10:54.532+0300    DEBUG   generic-jdbc-query-executor housekeeper com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Pool stats (total=13, active=0, idle=13, waiting=0)
2016-09-26T04:10:54.912+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@bd3ce29
2016-09-26T04:10:54.915+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@5142123
2016-09-26T04:10:54.922+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@12ba559
2016-09-26T04:10:54.926+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@44292b18
2016-09-26T04:10:54.929+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@519e0732
2016-09-26T04:10:54.933+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@7527291b
2016-09-26T04:10:54.938+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@297a5b8b
2016-09-26T04:10:54.946+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@2406d457
2016-09-26T04:10:54.953+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@500f072
2016-09-26T04:10:54.959+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@49a36e38
2016-09-26T04:10:54.963+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@327ed284
2016-09-26T04:10:54.968+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@20480cf0
2016-09-26T04:10:54.973+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@22c9ebeb
2016-09-26T04:10:54.977+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@70c99324
2016-09-26T04:10:54.981+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@3467954b
2016-09-26T04:10:54.991+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@542820bb
2016-09-26T04:10:54.995+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@39d31f66
2016-09-26T04:10:54.995+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - After adding stats (total=30, active=0, idle=30, waiting=0)
2016-09-26T04:10:55.580+0300    DEBUG   generic-jdbc-query-executor housekeeper com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Pool stats (total=1, active=1, idle=0, waiting=0)
2016-09-26T04:11:22.417+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@afb89ed: (connection has passed maxLifetime)
2016-09-26T04:11:22.462+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@53b998e8: (connection has passed maxLifetime)
2016-09-26T04:11:22.552+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@430d2704: (connection has passed maxLifetime)
2016-09-26T04:11:22.559+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@6b3fa525: (connection has passed maxLifetime)
2016-09-26T04:11:22.843+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@381d9ec8: (connection has passed maxLifetime)
2016-09-26T04:11:22.875+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@2a9ab87: (connection has passed maxLifetime)
2016-09-26T04:11:22.923+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@6651d141: (connection has passed maxLifetime)
2016-09-26T04:11:23.011+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@5985301a: (connection has passed maxLifetime)
2016-09-26T04:11:23.084+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@182e6d4f: (connection has passed maxLifetime)
2016-09-26T04:11:23.103+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@5c400805: (connection has passed maxLifetime)
2016-09-26T04:11:23.121+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@1f152ca1: (connection has passed maxLifetime)
2016-09-26T04:11:23.135+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@6ade7c61: (connection has passed maxLifetime)
2016-09-26T04:11:23.191+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@27fcc597: (connection has passed maxLifetime)
2016-09-26T04:11:23.192+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@53a29030: (connection has passed maxLifetime)
2016-09-26T04:11:23.413+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@60d54bdf: (connection has passed maxLifetime)
2016-09-26T04:11:23.419+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@579d1e1b: (connection has passed maxLifetime)
2016-09-26T04:11:23.479+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@72c17159: (connection has passed maxLifetime)
2016-09-26T04:11:23.514+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@4abec178: (connection has passed maxLifetime)
2016-09-26T04:11:23.547+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@54fdf8da: (connection has passed maxLifetime)
2016-09-26T04:11:23.573+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@14929c4e: (connection has passed maxLifetime)
2016-09-26T04:11:23.643+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@7b9c29d6: (connection has passed maxLifetime)
2016-09-26T04:11:23.651+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@405839d7: (connection has passed maxLifetime)
2016-09-26T04:11:23.724+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@5fc04255: (connection has passed maxLifetime)
2016-09-26T04:11:23.729+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@648185fb: (connection has passed maxLifetime)
2016-09-26T04:11:23.796+0300    DEBUG   generic-jdbc-query-executor housekeeper com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Pool stats (total=13, active=0, idle=13, waiting=0)
2016-09-26T04:11:23.801+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@290c69bc
2016-09-26T04:11:23.804+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@204d981
2016-09-26T04:11:23.808+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@de20e9b
2016-09-26T04:11:23.813+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@40dba463
2016-09-26T04:11:23.828+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@57547a64
2016-09-26T04:11:23.837+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@9bc33db
2016-09-26T04:11:23.840+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@3d80c221
2016-09-26T04:11:23.844+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@7f3f934f
2016-09-26T04:11:23.856+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@1de109ec
2016-09-26T04:11:23.859+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@608d54cc
2016-09-26T04:11:23.862+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@6e4d4c63
2016-09-26T04:11:23.862+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@592c8ab5: (connection has passed maxLifetime)
2016-09-26T04:11:23.868+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@25dbd066
2016-09-26T04:11:23.873+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@7622d9e1
2016-09-26T04:11:23.878+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@2d745360
2016-09-26T04:11:23.900+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@37fac2a9
2016-09-26T04:11:23.910+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@5fadf80a
2016-09-26T04:11:23.913+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@4968d75f
2016-09-26T04:11:23.914+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - After adding stats (total=29, active=0, idle=29, waiting=0)
2016-09-26T04:11:23.958+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@16a01420: (connection has passed maxLifetime)
2016-09-26T04:11:24.016+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@5c311aba: (connection has passed maxLifetime)
2016-09-26T04:11:24.159+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@1da0a4e4: (connection has passed maxLifetime)
2016-09-26T04:11:24.292+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@768feb6f: (connection has passed maxLifetime)
2016-09-26T04:11:24.432+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@6098e248: (connection has passed maxLifetime)
2016-09-26T04:11:24.535+0300    DEBUG   generic-jdbc-query-executor housekeeper com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Pool stats (total=18, active=0, idle=18, waiting=0)
2016-09-26T04:11:24.538+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@595afb40: (connection has passed maxLifetime)
2016-09-26T04:11:24.540+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@292f2182
2016-09-26T04:11:24.545+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@27e8dad7
2016-09-26T04:11:24.549+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@23d74af2
2016-09-26T04:11:24.553+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@66d5d0ec
2016-09-26T04:11:24.559+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@36a6de85
2016-09-26T04:11:24.562+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@577994e4
2016-09-26T04:11:24.566+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@4ee0c423
2016-09-26T04:11:24.569+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@6fa2fa3
2016-09-26T04:11:24.573+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@6288b0f
2016-09-26T04:11:24.577+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@1d6df19a
2016-09-26T04:11:24.580+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@55e4efc8
2016-09-26T04:11:24.585+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@2b455e0
2016-09-26T04:11:24.585+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - After adding stats (total=29, active=0, idle=29, waiting=0)
2016-09-26T04:11:25.584+0300    DEBUG   generic-jdbc-query-executor housekeeper com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Pool stats (total=1, active=1, idle=0, waiting=0)
2016-09-26T04:11:52.402+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@59dd925e: (connection has passed maxLifetime)
2016-09-26T04:11:52.719+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@70368c00: (connection has passed maxLifetime)
2016-09-26T04:11:52.773+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@11c44c6c: (connection has passed maxLifetime)
2016-09-26T04:11:52.821+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@76330d21: (connection has passed maxLifetime)
2016-09-26T04:11:52.856+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@6d6a8686: (connection has passed maxLifetime)
2016-09-26T04:11:53.218+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@2f32bd4d: (connection has passed maxLifetime)
2016-09-26T04:11:53.224+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@18198fd: (connection has passed maxLifetime)
2016-09-26T04:11:53.260+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@4814f6b: (connection has passed maxLifetime)
2016-09-26T04:11:53.418+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@7c98fb4b: (connection has passed maxLifetime)
2016-09-26T04:11:53.493+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@2063d13: (connection has passed maxLifetime)
2016-09-26T04:11:53.557+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@500f072: (connection has passed maxLifetime)
2016-09-26T04:11:53.567+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@5142123: (connection has passed maxLifetime)
2016-09-26T04:11:53.579+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@44292b18: (connection has passed maxLifetime)
2016-09-26T04:11:53.697+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@1e09997a: (connection has passed maxLifetime)
2016-09-26T04:11:53.748+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@69c4ec4: (connection has passed maxLifetime)
2016-09-26T04:11:53.799+0300    DEBUG   generic-jdbc-query-executor housekeeper com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Pool stats (total=17, active=0, idle=17, waiting=0)
2016-09-26T04:11:53.803+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@1716977f
2016-09-26T04:11:53.808+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@3fa47f6e
2016-09-26T04:11:53.812+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@486fbbf2
2016-09-26T04:11:53.823+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@1392a090
2016-09-26T04:11:53.837+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@7b44d8b4
2016-09-26T04:11:53.842+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@2f6843cb
2016-09-26T04:11:53.849+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@455d3860
2016-09-26T04:11:53.854+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@d0d0e14
2016-09-26T04:11:53.862+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@d436471
2016-09-26T04:11:53.869+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@3ac1cf2f
2016-09-26T04:11:53.875+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@36906ce6
2016-09-26T04:11:53.891+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@a606d8b
2016-09-26T04:11:53.891+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@3467954b: (connection has passed maxLifetime)
2016-09-26T04:11:53.896+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@3766ef68
2016-09-26T04:11:53.896+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - After adding stats (total=30, active=0, idle=30, waiting=0)
2016-09-26T04:11:54.013+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@12ba559: (connection has passed maxLifetime)
2016-09-26T04:11:54.088+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@70c99324: (connection has passed maxLifetime)
2016-09-26T04:11:54.139+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@39d31f66: (connection has passed maxLifetime)
2016-09-26T04:11:54.227+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@519e0732: (connection has passed maxLifetime)
2016-09-26T04:11:54.304+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@49a36e38: (connection has passed maxLifetime)
2016-09-26T04:11:54.445+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@542820bb: (connection has passed maxLifetime)
2016-09-26T04:11:54.537+0300    DEBUG   generic-jdbc-query-executor housekeeper com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Pool stats (total=19, active=0, idle=19, waiting=0)
2016-09-26T04:11:54.542+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@b21dd0e
2016-09-26T04:11:54.548+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@327ed284: (connection has passed maxLifetime)
2016-09-26T04:11:54.549+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@5e6221a
2016-09-26T04:11:54.553+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@1d9b67e
2016-09-26T04:11:54.560+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@44db2821
2016-09-26T04:11:54.568+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@41175419
2016-09-26T04:11:54.575+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@755548d3
2016-09-26T04:11:54.583+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@39736a41
2016-09-26T04:11:54.588+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@24f4d76e
2016-09-26T04:11:54.593+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@5f38f875
2016-09-26T04:11:54.599+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@f43091
2016-09-26T04:11:54.605+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection org.postgresql.jdbc4.Jdbc4Connection@227763ac
2016-09-26T04:11:54.605+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - After adding stats (total=29, active=0, idle=29, waiting=0)
2016-09-26T04:11:54.814+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@2406d457: (connection has passed maxLifetime)
2016-09-26T04:11:54.856+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@bd3ce29: (connection has passed maxLifetime)
2016-09-26T04:11:54.857+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@22c9ebeb: (connection has passed maxLifetime)
2016-09-26T04:11:54.873+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@297a5b8b: (connection has passed maxLifetime)
2016-09-26T04:11:54.906+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@7527291b: (connection has passed maxLifetime)
2016-09-26T04:11:54.959+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@20480cf0: (connection has passed maxLifetime)
2016-09-26T04:11:55.589+0300    DEBUG   generic-jdbc-query-executor housekeeper com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Pool stats (total=1, active=1, idle=0, waiting=0)
2016-09-26T04:11:57.533+0300    INFO    nioEventLoopGroup-5-1   io.netty.handler.logging.LoggingHandler [id: 0xeb6978f0, /0:0:0:0:0:0:0:0:9999] RECEIVED: [id: 0x08f3b9c2, /127.0.0.1:55626 => /127.0.0.1:9999]
2016-09-26T04:12:25.280+0300    DEBUG   generic-jdbc-query-executor housekeeper com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Pool stats (total=13, active=0, idle=13, waiting=0)
2016-09-26T04:12:25.280+0300    DEBUG   generic-jdbc-query-executor housekeeper com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Pool stats (total=11, active=0, idle=11, waiting=0)
2016-09-26T04:12:25.280+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@66d5d0ec: (connection has passed maxLifetime)
2016-09-26T04:12:25.288+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@6fa2fa3: (connection has passed maxLifetime)
2016-09-26T04:12:25.288+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@3d80c221: (connection has passed maxLifetime)
2016-09-26T04:12:25.296+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@55e4efc8: (connection has passed maxLifetime)
2016-09-26T04:12:25.296+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@4968d75f: (connection has passed maxLifetime)
2016-09-26T04:12:25.280+0300    DEBUG   nioEventLoopGroup-4-7   com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Timeout failure stats (total=1, active=1, idle=0, waiting=0)
2016-09-26T04:12:25.306+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@36a6de85: (connection has passed maxLifetime)
2016-09-26T04:12:25.306+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@de20e9b: (connection has passed maxLifetime)
2016-09-26T04:12:25.308+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@292f2182: (connection has passed maxLifetime)
2016-09-26T04:12:25.308+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@204d981: (connection has passed maxLifetime)
2016-09-26T04:12:25.312+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@1d6df19a: (connection has passed maxLifetime)
2016-09-26T04:12:25.312+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@290c69bc: (connection has passed maxLifetime)
2016-09-26T04:12:25.314+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@7622d9e1: (connection has passed maxLifetime)
2016-09-26T04:12:25.314+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@1de109ec: (connection has passed maxLifetime)
2016-09-26T04:12:25.314+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@27e8dad7: (connection has passed maxLifetime)
2016-09-26T04:12:25.314+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@57547a64: (connection has passed maxLifetime)
2016-09-26T04:12:25.316+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@37fac2a9: (connection has passed maxLifetime)
2016-09-26T04:12:25.316+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@6288b0f: (connection has passed maxLifetime)
2016-09-26T04:12:25.322+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@7f3f934f: (connection has passed maxLifetime)
2016-09-26T04:12:25.322+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@4ee0c423: (connection has passed maxLifetime)
2016-09-26T04:12:25.322+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@25dbd066: (connection has passed maxLifetime)
2016-09-26T04:12:25.323+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@2b455e0: (connection has passed maxLifetime)
2016-09-26T04:12:25.323+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@9bc33db: (connection has passed maxLifetime)
2016-09-26T04:12:25.328+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@23d74af2: (connection has passed maxLifetime)
2016-09-26T04:12:25.328+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@6e4d4c63: (connection has passed maxLifetime)
2016-09-26T04:12:25.331+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@40dba463: (connection has passed maxLifetime)
2016-09-26T04:12:25.331+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@577994e4: (connection has passed maxLifetime)
2016-09-26T04:12:25.334+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@608d54cc: (connection has passed maxLifetime)
2016-09-26T04:12:25.334+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@2d745360: (connection has passed maxLifetime)
2016-09-26T04:12:25.334+0300    DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@5fadf80a: (connection has passed maxLifetime)
2016-09-26T04:12:25.334+0300    DEBUG   generic-jdbc-query-executor connection adder    com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Added connection com.mysql.jdbc.JDBC4Connection@54b4105e
2016-09-26T04:12:25.322+0300    ERROR   nioEventLoopGroup-4-7   org.rakam.server.http.HttpServer    An uncaught exception raised while processing request.
org.skife.jdbi.v2.exceptions.UnableToObtainConnectionException: java.sql.SQLTransientConnectionException: generic-jdbc-query-executor - Connection is not available, request timed out after 27780ms.
    at org.skife.jdbi.v2.DBI.open(DBI.java:230)
    at org.rakam.ui.user.WebUserService.getUser(WebUserService.java:631)
    at org.rakam.ui.user.WebUserHttpService.me(WebUserHttpService.java:297)
    at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
    at org.rakam.server.http.JsonParametrizedRequestHandler.handleInternal(JsonParametrizedRequestHandler.java:126)
    at org.rakam.server.http.JsonParametrizedRequestHandler.handle(JsonParametrizedRequestHandler.java:93)
    at org.rakam.server.http.RouteMatcher.handle(RouteMatcher.java:59)
    at org.rakam.server.http.HttpServerHandler.channelRead(HttpServerHandler.java:75)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:276)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:263)
    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLTransientConnectionException: generic-jdbc-query-executor - Connection is not available, request timed out after 27780ms.
    at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:548)
    at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:186)
    at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145)
    at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:83)
    at org.rakam.analysis.JDBCPoolDataSource.getConnection(JDBCPoolDataSource.java:73)
    at org.skife.jdbi.v2.DataSourceConnectionFactory.openConnection(DataSourceConnectionFactory.java:36)
    at org.skife.jdbi.v2.DBI.open(DBI.java:212)
    ... 23 more

In fact, WebUserService only borrow one connection and uses it to run two separate queries so even if there is only one available connection, it should work in theory.

Here is the code: https://github.com/rakam-io/rakam/blob/562f74accd870641f6d1ee56d5dad45e4f56d142/rakam-ui/src/main/java/org/rakam/ui/user/WebUserService.java#L631

@buremba buremba changed the title Connection is not available even if there idle connections. Connection is not available error even if there idle connections. Sep 26, 2016
@brettwooldridge
Copy link
Owner

@buremba Can you post the output of the pool when it starts? I would like to see the pool configuration that is logged.

Somehow it looks like the HikariCP is applying a maxLifetime of 1 minute (!) to pool connections. When I trace the life of a single connection, I see:

04:09:24.528 DEBUG ... com.zaxxer.hikari.pool.HikariPool ... - Added connection org.postgresql.jdbc4.Jdbc4Connection@5d7e4b1e
...
04:10:23.310 DEBUG ... com.zaxxer.hikari.pool.PoolBase   ... - Closing connection org.postgresql.jdbc4.Jdbc4Connection@5d7e4b1e: (connection has passed maxLifetime)

@brettwooldridge
Copy link
Owner

@buremba That is a super-confusing log. I see these two adjacent log statements:

04:12:25.280 DEBUG   generic-jdbc-query-executor housekeeper com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Pool stats (total=13, active=0, idle=13, waiting=0)
04:12:25.280 DEBUG   generic-jdbc-query-executor housekeeper com.zaxxer.hikari.pool.HikariPool   generic-jdbc-query-executor - Pool stats (total=11, active=0, idle=11, waiting=0)

Which I thought should be theoretically impossible, because the HouseKeeper is scheduled to run every thirty-seconds -- houseKeepingExecutorService.scheduleWithFixedDelay(...). But then I noticed what appears to be two pools running ... one for postgresql and one for mysql ... but with the same pool name (!)...

04:12:25.288 DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection org.postgresql.jdbc4.Jdbc4Connection@6fa2fa3: (connection has passed maxLifetime)
04:12:25.288 DEBUG   generic-jdbc-query-executor connection closer   com.zaxxer.hikari.pool.PoolBase generic-jdbc-query-executor - Closing connection com.mysql.jdbc.JDBC4Connection@3d80c221: (connection has passed maxLifetime)

If that is correct, dude, you have to specify that kind of information when you open an issue.

Rakam.io seems to set a fixed pool name. That is really bad practice. For one, it means that you cannot enable JMX, because two pools with the same name cannot be registered.

What is going on? How is this thing configured?

@buremba
Copy link
Author

buremba commented Sep 26, 2016

Hey @brettwooldridge, you're right, that's a bad practice. We realized that issue recently and tried to debug it. 1 total connection is not normal so I put a breakpoint where Hikari tries to throw the timeout error and realized that both log entries for 30 connections and 1 connection belong to Postgresql.

I actually tried a bunch of different configurations for lifetime and leak detection threshold and none of them worked. Most probably maximum lifetime value was 60000 in this example (I tried 60s because I thought that the socket connection might become unreliable after some time) so I was wrong when I said it uses the default configuration. It's likely that Hikari tries to connect the database for 60 seconds and then expires the connection. However, when I disable Hikari it works smooth and it shouldn't take 60s for Hikari to connect the database.

OK, I fixed the pool name issue now and restarted the server. I will paste the output when it throws the error again.

@brettwooldridge
Copy link
Owner

@buremba At least with unique pool names reading the log will be immeasurably easier. The default connectionTimeout is 30 seconds, so unless the driver is not honoring the setting (HikariCP calls setLoginTimeout() on the Driver/DataSource), then no connection attempt should exceed that.

@brettwooldridge
Copy link
Owner

brettwooldridge commented Sep 26, 2016

@buremba I'd really like to see a log that starts from when the pool is initialized, and continues to where the error is encountered. If you are on Linux, you should be able to grep the log file using the pool name to get only the lines that relate to the pool with the failure. You can zip it and attach it here if it is too large.

@buremba
Copy link
Author

buremba commented Sep 26, 2016

OK, I managed to get the same error. However since the output was stdout and my terminal has limited buffer, it trimmed the logs that is written when the process is started. :(

I didn't change any configuration and restarted restarted the process again so the configuration is the same as previous run:

2016-09-27T00:16:16.603+0300    INFO    main    io.airlift.log.Logging  Logging to stderr
2016-09-27T00:16:16.618+0300    INFO    main    org.rakam.bootstrap.Bootstrap   Loading configuration
2016-09-27T00:16:17.272+0300    INFO    main    org.rakam.bootstrap.Bootstrap   Initializing logging
2016-09-27T00:16:18.200+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  root@jdbc:mysql://127.0.0.1:3306/presto - configuration:
2016-09-27T00:16:18.209+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  allowPoolSuspension.............false
2016-09-27T00:16:18.210+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  autoCommit......................true
2016-09-27T00:16:18.210+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  catalog.........................null
2016-09-27T00:16:18.210+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  connectionInitSql...............null
2016-09-27T00:16:18.210+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  connectionTestQuery.............null
2016-09-27T00:16:18.211+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  connectionTimeout...............30000
2016-09-27T00:16:18.212+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  dataSource......................null
2016-09-27T00:16:18.212+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  dataSourceClassName.............null
2016-09-27T00:16:18.212+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  dataSourceJNDI..................null
2016-09-27T00:16:18.214+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  dataSourceProperties............{password=<masked>}
2016-09-27T00:16:18.215+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  driverClassName.................null
2016-09-27T00:16:18.215+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  healthCheckProperties...........{}
2016-09-27T00:16:18.215+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  healthCheckRegistry.............null
2016-09-27T00:16:18.215+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  idleTimeout.....................600000
2016-09-27T00:16:18.215+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  initializationFailFast..........true
2016-09-27T00:16:18.216+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  isolateInternalQueries..........false
2016-09-27T00:16:18.216+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  jdbc4ConnectionTest.............false
2016-09-27T00:16:18.216+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  jdbcUrl........................."jdbc:mysql://127.0.0.1:3306/presto"
2016-09-27T00:16:18.216+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  leakDetectionThreshold..........0
2016-09-27T00:16:18.216+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  maxLifetime.....................1800000
2016-09-27T00:16:18.216+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  maximumPoolSize.................30
2016-09-27T00:16:18.217+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  metricRegistry..................null
2016-09-27T00:16:18.217+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  metricsTrackerFactory...........null
2016-09-27T00:16:18.217+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  minimumIdle.....................30
2016-09-27T00:16:18.217+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  password........................<masked>
2016-09-27T00:16:18.217+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  poolName........................"root@jdbc:mysql://127.0.0.1:3306/presto"
2016-09-27T00:16:18.217+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  readOnly........................false
2016-09-27T00:16:18.217+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  registerMbeans..................false
2016-09-27T00:16:18.218+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  scheduledExecutorService........null
2016-09-27T00:16:18.218+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  threadFactory...................null
2016-09-27T00:16:18.218+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  transactionIsolation............null
2016-09-27T00:16:18.218+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  username........................"root"
2016-09-27T00:16:18.218+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  validationTimeout...............5000
2016-09-27T00:16:18.223+0300    INFO    main    com.zaxxer.hikari.HikariDataSource  root@jdbc:mysql://127.0.0.1:3306/presto - Started.
2016-09-27T00:16:20.345+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto housekeeper com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Pool stats (total=0, active=0, idle=0, waiting=0)
2016-09-27T00:16:20.358+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  rakam@jdbc:postgresql://rakam-prod-rakam.chmo5weps9zn.us-east-1.rds.amazonaws.com:5432/rakam - configuration:
2016-09-27T00:16:20.360+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  allowPoolSuspension.............false
2016-09-27T00:16:20.360+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  autoCommit......................true
2016-09-27T00:16:20.360+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  catalog.........................null
2016-09-27T00:16:20.360+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  connectionInitSql...............null
2016-09-27T00:16:20.360+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  connectionTestQuery.............null
2016-09-27T00:16:20.360+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  connectionTimeout...............30000
2016-09-27T00:16:20.360+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  dataSource......................null
2016-09-27T00:16:20.361+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  dataSourceClassName.............null
2016-09-27T00:16:20.361+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  dataSourceJNDI..................null
2016-09-27T00:16:20.361+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  dataSourceProperties............{password=<masked>}
2016-09-27T00:16:20.361+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  driverClassName.................null
2016-09-27T00:16:20.361+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  healthCheckProperties...........{}
2016-09-27T00:16:20.361+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  healthCheckRegistry.............null
2016-09-27T00:16:20.361+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  idleTimeout.....................600000
2016-09-27T00:16:20.362+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  initializationFailFast..........true
2016-09-27T00:16:20.362+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  isolateInternalQueries..........false
2016-09-27T00:16:20.362+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  jdbc4ConnectionTest.............false
2016-09-27T00:16:20.362+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  jdbcUrl........................."jdbc:postgresql://rakam-prod-rakam.chmo5weps9zn.us-east-1.rds.amazonaws.com:5432/rakam"
2016-09-27T00:16:20.363+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  leakDetectionThreshold..........0
2016-09-27T00:16:20.363+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  maxLifetime.....................1800000
2016-09-27T00:16:20.363+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  maximumPoolSize.................30
2016-09-27T00:16:20.363+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  metricRegistry..................null
2016-09-27T00:16:20.364+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  metricsTrackerFactory...........null
2016-09-27T00:16:20.364+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  minimumIdle.....................30
2016-09-27T00:16:20.364+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  password........................<masked>
2016-09-27T00:16:20.364+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  poolName........................"rakam@jdbc:postgresql://rakam-prod-rakam.chmo5weps9zn.us-east-1.rds.amazonaws.com:5432/rakam"
2016-09-27T00:16:20.364+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  readOnly........................false
2016-09-27T00:16:20.364+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  registerMbeans..................false
2016-09-27T00:16:20.364+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  scheduledExecutorService........null
2016-09-27T00:16:20.364+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  threadFactory...................null
2016-09-27T00:16:20.364+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  transactionIsolation............null
2016-09-27T00:16:20.365+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  username........................"rakam"
2016-09-27T00:16:20.365+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  validationTimeout...............5000
2016-09-27T00:16:20.365+0300    INFO    main    com.zaxxer.hikari.HikariDataSource  rakam@jdbc:postgresql://rakam-prod-rakam.chmo5weps9zn.us-east-1.rds.amazonaws.com:5432/rakam - Started.

And here is the output: https://gist.github.com/buremba/a34e80a306909e3d60635ec9d21ceb0b

This the full output and I didn't use grep since it looks for the strings line by line AFAIK. When I simply run cat app.log | grep rakam@ it removed the stacktrace lines in the log.

The strange thing is that it worked fine for 5 hours when the computer is on, when I put the computer in sleep mode and open it after an hour, the error occurred again. Hope it helps.

@brettwooldridge
Copy link
Owner

@buremba I'm confused. The configuration above has two pool configurations:

root@jdbc:mysql://127.0.0.1:3306/presto
rakam@jdbc:postgresql://rakam-prod-rakam.chmo5weps9zn.us-east-1.rds.amazonaws.com:5432/rakam

But the log (gist) contains three:

18:41:10.404 DEBUG rakam@jdbc:postgresql://rakam-prod-rakam.chmo5weps9zn.us-east-1.rds.amazonaws.com:5432/rakam housekeeper ... - Pool stats (total=30, active=0, idle=30, waiting=0)
18:41:39.245 DEBUG root@jdbc:mysql://127.0.0.1:3306/presto housekeeper ... - Pool stats (total=30, active=0, idle=30, waiting=0)
18:42:10.340 DEBUG buremba@jdbc:postgresql://127.0.0.1:5432/rakam housekeeper ... - Pool stats (total=30, active=0, idle=30, waiting=0)

@buremba
Copy link
Author

buremba commented Sep 27, 2016

The last one initialized after them but I didn't put in the logs that I pasted. Sorry for the confusion. I focused on the last entries of the full log, as you see there are 30 idle connection for that pool but we still get timeout errors. Also it happened again even if I didn't put the computer in sleep mode. I'm not sure what might cause this issue but it seems that it cannot borrow a connection from the pool.

2016-09-27T03:35:29.326+0300    INFO    main    io.airlift.log.Logging  Logging to stderr
2016-09-27T03:35:29.332+0300    INFO    main    org.rakam.bootstrap.Bootstrap   Loading configuration
2016-09-27T03:35:29.977+0300    INFO    main    org.rakam.bootstrap.Bootstrap   Initializing logging
2016-09-27T03:35:30.580+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  root@jdbc:mysql://127.0.0.1:3306/presto - configuration:
2016-09-27T03:35:30.588+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  allowPoolSuspension.............false
2016-09-27T03:35:30.588+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  autoCommit......................true
2016-09-27T03:35:30.589+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  catalog.........................null
2016-09-27T03:35:30.589+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  connectionInitSql...............null
2016-09-27T03:35:30.589+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  connectionTestQuery.............null
2016-09-27T03:35:30.589+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  connectionTimeout...............30000
2016-09-27T03:35:30.589+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  dataSource......................null
2016-09-27T03:35:30.589+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  dataSourceClassName.............null
2016-09-27T03:35:30.589+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  dataSourceJNDI..................null
2016-09-27T03:35:30.591+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  dataSourceProperties............{password=<masked>}
2016-09-27T03:35:30.591+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  driverClassName.................null
2016-09-27T03:35:30.591+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  healthCheckProperties...........{}
2016-09-27T03:35:30.591+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  healthCheckRegistry.............null
2016-09-27T03:35:30.591+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  idleTimeout.....................600000
2016-09-27T03:35:30.591+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  initializationFailFast..........true
2016-09-27T03:35:30.591+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  isolateInternalQueries..........false
2016-09-27T03:35:30.592+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  jdbc4ConnectionTest.............false
2016-09-27T03:35:30.592+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  jdbcUrl........................."jdbc:mysql://127.0.0.1:3306/presto"
2016-09-27T03:35:30.592+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  leakDetectionThreshold..........0
2016-09-27T03:35:30.592+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  maxLifetime.....................1800000
2016-09-27T03:35:30.592+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  maximumPoolSize.................30
2016-09-27T03:35:30.592+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  metricRegistry..................null
2016-09-27T03:35:30.592+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  metricsTrackerFactory...........null
2016-09-27T03:35:30.592+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  minimumIdle.....................30
2016-09-27T03:35:30.593+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  password........................<masked>
2016-09-27T03:35:30.593+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  poolName........................"root@jdbc:mysql://127.0.0.1:3306/presto"
2016-09-27T03:35:30.593+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  readOnly........................false
2016-09-27T03:35:30.593+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  registerMbeans..................false
2016-09-27T03:35:30.593+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  scheduledExecutorService........null
2016-09-27T03:35:30.593+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  threadFactory...................null
2016-09-27T03:35:30.593+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  transactionIsolation............null
2016-09-27T03:35:30.593+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  username........................"root"
2016-09-27T03:35:30.594+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  validationTimeout...............5000
2016-09-27T03:35:30.596+0300    INFO    main    com.zaxxer.hikari.HikariDataSource  root@jdbc:mysql://127.0.0.1:3306/presto - Started.
2016-09-27T03:35:32.479+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto housekeeper com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Pool stats (total=0, active=0, idle=0, waiting=0)
2016-09-27T03:35:32.493+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  rakam@jdbc:postgresql://rakam-prod-rakam.chmo5weps9zn.us-east-1.rds.amazonaws.com:5432/rakam - configuration:
2016-09-27T03:35:32.495+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  allowPoolSuspension.............false
2016-09-27T03:35:32.495+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  autoCommit......................true
2016-09-27T03:35:32.495+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  catalog.........................null
2016-09-27T03:35:32.495+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  connectionInitSql...............null
2016-09-27T03:35:32.495+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  connectionTestQuery.............null
2016-09-27T03:35:32.495+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  connectionTimeout...............30000
2016-09-27T03:35:32.495+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  dataSource......................null
2016-09-27T03:35:32.496+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  dataSourceClassName.............null
2016-09-27T03:35:32.496+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  dataSourceJNDI..................null
2016-09-27T03:35:32.496+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  dataSourceProperties............{password=<masked>}
2016-09-27T03:35:32.496+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  driverClassName.................null
2016-09-27T03:35:32.496+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  healthCheckProperties...........{}
2016-09-27T03:35:32.496+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  healthCheckRegistry.............null
2016-09-27T03:35:32.496+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  idleTimeout.....................600000
2016-09-27T03:35:32.496+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  initializationFailFast..........true
2016-09-27T03:35:32.496+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  isolateInternalQueries..........false
2016-09-27T03:35:32.496+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  jdbc4ConnectionTest.............false
2016-09-27T03:35:32.497+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  jdbcUrl........................."jdbc:postgresql://rakam-prod-rakam.chmo5weps9zn.us-east-1.rds.amazonaws.com:5432/rakam"
2016-09-27T03:35:32.497+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  leakDetectionThreshold..........0
2016-09-27T03:35:32.497+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  maxLifetime.....................1800000
2016-09-27T03:35:32.497+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  maximumPoolSize.................30
2016-09-27T03:35:32.497+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  metricRegistry..................null
2016-09-27T03:35:32.497+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  metricsTrackerFactory...........null
2016-09-27T03:35:32.497+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  minimumIdle.....................30
2016-09-27T03:35:32.497+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  password........................<masked>
2016-09-27T03:35:32.497+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  poolName........................"rakam@jdbc:postgresql://rakam-prod-rakam.chmo5weps9zn.us-east-1.rds.amazonaws.com:5432/rakam"
2016-09-27T03:35:32.497+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  readOnly........................false
2016-09-27T03:35:32.498+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  registerMbeans..................false
2016-09-27T03:35:32.498+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  scheduledExecutorService........null
2016-09-27T03:35:32.498+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  threadFactory...................null
2016-09-27T03:35:32.498+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  transactionIsolation............null
2016-09-27T03:35:32.498+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  username........................"rakam"
2016-09-27T03:35:32.498+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  validationTimeout...............5000
2016-09-27T03:35:32.498+0300    INFO    main    com.zaxxer.hikari.HikariDataSource  rakam@jdbc:postgresql://rakam-prod-rakam.chmo5weps9zn.us-east-1.rds.amazonaws.com:5432/rakam - Started.
2016-09-27T03:35:32.535+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@5ddfcd8e
2016-09-27T03:35:32.564+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@6aeb923a
2016-09-27T03:35:32.575+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@77110505
2016-09-27T03:35:32.584+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@8bddcb3
2016-09-27T03:35:32.599+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@7131bb16
2016-09-27T03:35:32.614+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@1845b6ac
2016-09-27T03:35:32.626+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@3e47ea2d
2016-09-27T03:35:32.632+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@c194538
2016-09-27T03:35:32.640+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@78762979
2016-09-27T03:35:32.652+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@329ffaab
2016-09-27T03:35:32.657+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@22f94e4b
2016-09-27T03:35:32.665+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@1aad0540
2016-09-27T03:35:32.672+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@4fb95f2c
2016-09-27T03:35:32.681+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@41c6baaf
2016-09-27T03:35:32.691+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@16c7581
2016-09-27T03:35:32.705+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@25102297
2016-09-27T03:35:32.719+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@1d5b6e05
2016-09-27T03:35:32.740+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@5ca839e3
2016-09-27T03:35:32.780+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@25ad908a
2016-09-27T03:35:32.794+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@4b1c9db6
2016-09-27T03:35:32.811+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@40361d80
2016-09-27T03:35:32.823+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@d1611a5
2016-09-27T03:35:32.828+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@1d3d86ca
2016-09-27T03:35:32.834+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@f6a8a1
2016-09-27T03:35:32.841+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@3d9602e7
2016-09-27T03:35:32.846+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@441d22f1
2016-09-27T03:35:32.853+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@3d738bd6
2016-09-27T03:35:32.859+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@73279d9f
2016-09-27T03:35:32.865+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@15654ff3
2016-09-27T03:35:32.874+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - Added connection com.mysql.jdbc.JDBC4Connection@740cc9d
2016-09-27T03:35:32.874+0300    DEBUG   root@jdbc:mysql://127.0.0.1:3306/presto connection adder    com.zaxxer.hikari.pool.HikariPool   root@jdbc:mysql://127.0.0.1:3306/presto - After adding stats (total=30, active=0, idle=30, waiting=0)
2016-09-27T03:35:33.358+0300    INFO    main    com.zaxxer.hikari.pool.PoolBase rakam@jdbc:postgresql://rakam-prod-rakam.chmo5weps9zn.us-east-1.rds.amazonaws.com:5432/rakam - Driver does not support get/set network timeout for connections. (Method org.postgresql.jdbc4.Jdbc4Connection.getNetworkTimeout() is not yet implemented.)
2016-09-27T03:35:33.733+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  buremba@jdbc:postgresql://127.0.0.1:5432/rakam - configuration:
2016-09-27T03:35:33.735+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  allowPoolSuspension.............false
2016-09-27T03:35:33.735+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  autoCommit......................true
2016-09-27T03:35:33.735+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  catalog.........................null
2016-09-27T03:35:33.735+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  connectionInitSql...............null
2016-09-27T03:35:33.735+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  connectionTestQuery.............null
2016-09-27T03:35:33.735+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  connectionTimeout...............30000
2016-09-27T03:35:33.735+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  dataSource......................null
2016-09-27T03:35:33.736+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  dataSourceClassName.............null
2016-09-27T03:35:33.736+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  dataSourceJNDI..................null
2016-09-27T03:35:33.736+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  dataSourceProperties............{password=<masked>}
2016-09-27T03:35:33.736+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  driverClassName.................null
2016-09-27T03:35:33.736+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  healthCheckProperties...........{}
2016-09-27T03:35:33.736+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  healthCheckRegistry.............null
2016-09-27T03:35:33.736+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  idleTimeout.....................600000
2016-09-27T03:35:33.736+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  initializationFailFast..........true
2016-09-27T03:35:33.736+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  isolateInternalQueries..........false
2016-09-27T03:35:33.736+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  jdbc4ConnectionTest.............false
2016-09-27T03:35:33.736+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  jdbcUrl........................."jdbc:postgresql://127.0.0.1:5432/rakam"
2016-09-27T03:35:33.736+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  leakDetectionThreshold..........0
2016-09-27T03:35:33.737+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  maxLifetime.....................1800000
2016-09-27T03:35:33.737+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  maximumPoolSize.................30
2016-09-27T03:35:33.737+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  metricRegistry..................null
2016-09-27T03:35:33.737+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  metricsTrackerFactory...........null
2016-09-27T03:35:33.737+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  minimumIdle.....................30
2016-09-27T03:35:33.737+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  password........................<masked>
2016-09-27T03:35:33.737+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  poolName........................"buremba@jdbc:postgresql://127.0.0.1:5432/rakam"
2016-09-27T03:35:33.737+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  readOnly........................false
2016-09-27T03:35:33.737+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  registerMbeans..................false
2016-09-27T03:35:33.737+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  scheduledExecutorService........null
2016-09-27T03:35:33.737+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  threadFactory...................null
2016-09-27T03:35:33.738+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  transactionIsolation............null
2016-09-27T03:35:33.738+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  username........................"buremba"
2016-09-27T03:35:33.738+0300    DEBUG   main    com.zaxxer.hikari.HikariConfig  validationTimeout...............5000
2016-09-27T03:35:33.738+0300    INFO    main    com.zaxxer.hikari.HikariDataSource  buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Started.
2016-09-27T03:35:33.781+0300    INFO    main    com.zaxxer.hikari.pool.PoolBase buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Driver does not support get/set network timeout for connections. (Method org.postgresql.jdbc4.Jdbc4Connection.getNetworkTimeout() is not yet implemented.)
2016-09-27T03:35:33.813+0300    DEBUG   rakam@jdbc:postgresql://rakam-prod-rakam.chmo5weps9zn.us-east-1.rds.amazonaws.com:5432/rakam housekeeper    com.zaxxer.hikari.pool.HikariPool   rakam@jdbc:postgresql://rakam-prod-rakam.chmo5weps9zn.us-east-1.rds.amazonaws.com:5432/rakam - Pool stats (total=0, active=0, idle=0, waiting=0)
2016-09-27T03:35:33.883+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam housekeeper  com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Pool stats (total=0, active=0, idle=0, waiting=0)
2016-09-27T03:35:33.928+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@20c0b496
2016-09-27T03:35:33.937+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@2e075efe
2016-09-27T03:35:33.961+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@2a38dfe6
2016-09-27T03:35:33.968+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@6f3a54c5
2016-09-27T03:35:33.978+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@4afb39b0
2016-09-27T03:35:34.002+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@5e1f7ebf
2016-09-27T03:35:34.012+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@32344d75
2016-09-27T03:35:34.022+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@f15ab8a
2016-09-27T03:35:34.032+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@1c303561
2016-09-27T03:35:34.042+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@39ff4ade
2016-09-27T03:35:34.053+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@2930e771
2016-09-27T03:35:34.066+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@b5b5026
2016-09-27T03:35:34.075+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@16ddf711
2016-09-27T03:35:34.084+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@5537e6ae
2016-09-27T03:35:34.101+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@7b928816
2016-09-27T03:35:34.108+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@2a992d22
2016-09-27T03:35:34.116+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@5305321f
2016-09-27T03:35:34.122+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@39edbc47
2016-09-27T03:35:34.127+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@56505154
2016-09-27T03:35:34.133+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@35439681
2016-09-27T03:35:34.142+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@4f2cdffc
2016-09-27T03:35:34.147+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@1b775dda
2016-09-27T03:35:34.157+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@4f797511
2016-09-27T03:35:34.171+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@662705e1
2016-09-27T03:35:34.180+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@4fdde5e6
2016-09-27T03:35:34.191+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@6ee5f84a
2016-09-27T03:35:34.209+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@6a30c97a
2016-09-27T03:35:34.221+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@3466d1d0
2016-09-27T03:35:34.233+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@1434e06f
2016-09-27T03:35:34.245+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - Added connection org.postgresql.jdbc4.Jdbc4Connection@55d26d0b
2016-09-27T03:35:34.246+0300    DEBUG   buremba@jdbc:postgresql://127.0.0.1:5432/rakam connection adder com.zaxxer.hikari.pool.HikariPool   buremba@jdbc:postgresql://127.0.0.1:5432/rakam - After adding stats (total=30, active=0, idle=30, waiting=0)

@brettwooldridge
Copy link
Owner

Your computer is clearly going to sleep ... this log shows a 12m 37s sleep:

21:26:48.701 WARN ... root@jdbc:mysql://127.0.0.1:3306/presto housekeeper ... - Thread starvation or clock leap detected (housekeeper delta=12m37s345ms).
...
21:26:55.656 WARN ... buremba@jdbc:postgresql://127.0.0.1:5432/rakam housekeeper ... - Thread starvation or clock leap detected (housekeeper delta=12m37s340ms).

Regarding the connection timeout, the last few lines of the log show all that you need to know:

23:51:07.778 WARN ... rakam@jdbc:postgresql://rakam-prod-rakam.chmo5weps9zn.us-east-1.rds.amazonaws.com:5432/rakam housekeeper ... - Thread starvation or clock leap detected (housekeeper delta=47s505ms).
...
23:51:07.909 ERROR ... nioEventLoopGroup-4-9    org.rakam.server.http.HttpServer    An uncaught exception raised while processing request.
org.skife.jdbi.v2.exceptions.UnableToObtainConnectionException: java.sql.SQLTransientConnectionException: rakam@jdbc:postgresql://rakam-prod-rakam.chmo5weps9zn.us-east-1.rds.amazonaws.com:5432/rakam - Connection is not available, request timed out after 135328ms.
    at org.skife.jdbi.v2.DBI.open(DBI.java:230)
    at org.rakam.ui.user.WebUserService.getUser(WebUserService.java:631)
    at org.rakam.ui.user.WebUserHttpService.me(WebUserHttpService.java:297)
    at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
    at org.rakam.server.http.JsonParametrizedRequestHandler.handleInternal(JsonParametrizedRequestHandler.java:126)
    at org.rakam.server.http.JsonParametrizedRequestHandler.handle(JsonParametrizedRequestHandler.java:93)
    at org.rakam.server.http.RouteMatcher.handle(RouteMatcher.java:59)
    at org.rakam.server.http.HttpServerHandler.channelRead(HttpServerHandler.java:75)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:276)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:263)
    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLTransientConnectionException: rakam@jdbc:postgresql://rakam-prod-rakam.chmo5weps9zn.us-east-1.rds.amazonaws.com:5432/rakam - Connection is not available, request timed out after 135328ms.
    at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:548)
    at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:186)
    at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145)
    at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:83)
    at org.rakam.analysis.JDBCPoolDataSource.getConnection(JDBCPoolDataSource.java:79)
    at org.skife.jdbi.v2.DataSourceConnectionFactory.openConnection(DataSourceConnectionFactory.java:36)
    at org.skife.jdbi.v2.DBI.open(DBI.java:212)
    ... 23 more

Timeout guarantees cannot be made when the CPU enters/exits power saving mode. The behavior of System.nanoTime() and System.currentTimeMillis() are unspecified in this case, and the result will depend on both the hardware and operating system.

@buremba
Copy link
Author

buremba commented Sep 27, 2016

Even if there is a clock leap after I opened the computer, the connection request is made when the computer is in active state and in that time period (135 seconds) the clock should work as expected so why we still suffer from clock leap?

Does it mean that if we put the computer in sleep mode and wake up the Hikari Pool becomes unusable?

@brettwooldridge
Copy link
Owner

HikariCP will not become unstable, it will recover from the clock shift, but HikariCP cannot guarantee that you will not get timeouts if the clock shifts.

HikariCP is designed for high-performance servers, not computers that go to sleep.

However, my bet is you are unlikely to find a pool that operates without error if the computer is going into and out of sleep mode. But I encourage you to find out for yourself.

@buremba
Copy link
Author

buremba commented Oct 8, 2016

OK, it seems reasonable design decision since we will get the most benefit from Hikari when it's used in high-performance servers but most of the developers test their applications in local environment which is usually a laptop that supports sleep mode and getting these errors in local may be misleading.
Therefore, you may consider adding a node to the documentation for this issue, it would be really helpful. BTW, Thanks for the support!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants