New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Connection is not available error even if there idle connections. #731
Comments
@buremba Can you post the output of the pool when it starts? I would like to see the pool configuration that is logged. Somehow it looks like the HikariCP is applying a
|
@buremba That is a super-confusing log. I see these two adjacent log statements:
Which I thought should be theoretically impossible, because the HouseKeeper is scheduled to run every thirty-seconds --
If that is correct, dude, you have to specify that kind of information when you open an issue. Rakam.io seems to set a fixed pool name. That is really bad practice. For one, it means that you cannot enable JMX, because two pools with the same name cannot be registered. What is going on? How is this thing configured? |
Hey @brettwooldridge, you're right, that's a bad practice. We realized that issue recently and tried to debug it. 1 total connection is not normal so I put a breakpoint where Hikari tries to throw the timeout error and realized that both log entries for 30 connections and 1 connection belong to Postgresql. I actually tried a bunch of different configurations for lifetime and leak detection threshold and none of them worked. Most probably maximum lifetime value was 60000 in this example (I tried 60s because I thought that the socket connection might become unreliable after some time) so I was wrong when I said it uses the default configuration. It's likely that Hikari tries to connect the database for 60 seconds and then expires the connection. However, when I disable Hikari it works smooth and it shouldn't take 60s for Hikari to connect the database. OK, I fixed the pool name issue now and restarted the server. I will paste the output when it throws the error again. |
@buremba At least with unique pool names reading the log will be immeasurably easier. The default |
@buremba I'd really like to see a log that starts from when the pool is initialized, and continues to where the error is encountered. If you are on Linux, you should be able to grep the log file using the pool name to get only the lines that relate to the pool with the failure. You can zip it and attach it here if it is too large. |
OK, I managed to get the same error. However since the output was stdout and my terminal has limited buffer, it trimmed the logs that is written when the process is started. :( I didn't change any configuration and restarted restarted the process again so the configuration is the same as previous run:
And here is the output: https://gist.github.com/buremba/a34e80a306909e3d60635ec9d21ceb0b This the full output and I didn't use grep since it looks for the strings line by line AFAIK. When I simply run The strange thing is that it worked fine for 5 hours when the computer is on, when I put the computer in sleep mode and open it after an hour, the error occurred again. Hope it helps. |
@buremba I'm confused. The configuration above has two pool configurations:
But the log (gist) contains three:
|
The last one initialized after them but I didn't put in the logs that I pasted. Sorry for the confusion. I focused on the last entries of the full log, as you see there are 30 idle connection for that pool but we still get timeout errors. Also it happened again even if I didn't put the computer in sleep mode. I'm not sure what might cause this issue but it seems that it cannot borrow a connection from the pool.
|
Your computer is clearly going to sleep ... this log shows a 12m 37s sleep:
Regarding the connection timeout, the last few lines of the log show all that you need to know:
Timeout guarantees cannot be made when the CPU enters/exits power saving mode. The behavior of |
Even if there is a clock leap after I opened the computer, the connection request is made when the computer is in active state and in that time period (135 seconds) the clock should work as expected so why we still suffer from clock leap? Does it mean that if we put the computer in sleep mode and wake up the Hikari Pool becomes unusable? |
HikariCP will not become unstable, it will recover from the clock shift, but HikariCP cannot guarantee that you will not get timeouts if the clock shifts. HikariCP is designed for high-performance servers, not computers that go to sleep. However, my bet is you are unlikely to find a pool that operates without error if the computer is going into and out of sleep mode. But I encourage you to find out for yourself. |
OK, it seems reasonable design decision since we will get the most benefit from Hikari when it's used in high-performance servers but most of the developers test their applications in local environment which is usually a laptop that supports sleep mode and getting these errors in local may be misleading. |
I use the default configuration and limit the maximum connection to 30.
When I start the application, it works fine but after some time, I'm getting timeout errors even if there is only one running query.
Normally, Hikari creates 30 connections but eventually it closes them in lifecycle process and keep only one connection in total.
Here is the logs:
In fact, WebUserService only borrow one connection and uses it to run two separate queries so even if there is only one available connection, it should work in theory.
Here is the code: https://github.com/rakam-io/rakam/blob/562f74accd870641f6d1ee56d5dad45e4f56d142/rakam-ui/src/main/java/org/rakam/ui/user/WebUserService.java#L631
The text was updated successfully, but these errors were encountered: