Skip to content

Connection pooling seems to be broken #24

Open
vhazrati opened this Issue Oct 31, 2011 · 13 comments

3 participants

@vhazrati

We have this object from which we are fetching our queryEvaluator

object MyDataSource {
val queryFactory = new SqlQueryFactory
val apachePoolingDatabaseFactory = new ApachePoolingDatabaseFactory(
"SELECT 1",
10,
10,
new Duration(1000),
new Duration(10),
true,
new Duration(30000))
val queryEvaluatorFactory = new StandardQueryEvaluatorFactory(apachePoolingDatabaseFactory, queryFactory)
val queryEvaluator = queryEvaluatorFactory(dbConfig.driverClassName, dbConfig.url, dbConfig.username, dbConfig.password)
}

We are not explicitly closing connections, assuming that it is handled. We seem to be running out of connections on our MySQL db.

We are using the following dependencies

libraryDependencies += "com.twitter" % "querulous" % "1.2.0-generic"

libraryDependencies += "mysql" % "mysql-connector-java" % "5.1.12"

@tsuna
tsuna commented Nov 7, 2011

It seems that I have the same problem. I'm using an AsyncQueryEvaluator that's pooling connections with ApachePoolingDatabase to a cluster of slaves. When doing a loadtest, my Finagle server quickly ran into java.io.IOException: Too many open files and lsof was showing that virtually all FDs were open connections to my various MySQL nodes.

I put up my code @ https://gist.github.com/1344664

edit: I'm using "com.twitter" % "querulous" % "2.5.0" in my sbt file.

@vhazrati
vhazrati commented Nov 8, 2011

tsuna are you using scala 2.8.x or 2.9x. Did you recompile querulous 2.5.0 to work with 2.9.x?

@tsuna
tsuna commented Nov 8, 2011

I'm using querulous-core-2.5.0.jar downloaded by SBT from Twitter's Maven repo, in conjunction with Scala 2.8.1, Finagle 1.9.5, mysql-connector-java 5.1.13, commons-pool-1.5.4.jar, and commons-dbcp-1.4.jar.

@tsuna
tsuna commented Nov 9, 2011

Halp! :)

Am I doing something obviously wrong? Maybe I shouldn't be getting a new QueryEvaluator for each query?

This issue is a show stopper for me right now :(

@vhazrati
vhazrati commented Nov 9, 2011

Same here, i am using Scala 2.9.1 but querulous 2.5.0 does not seem to work with that and i am forced to use 1.2.0-generic. I have posted on stackoverflow as well but awaiting a response http://stackoverflow.com/questions/7962766/querulous-connection-pooling-does-not-seem-to-work

@vhazrati
vhazrati commented Nov 9, 2011

also i seem to be getting the same MyDataSource.queryEvaluator still the issue.

@tsuna
tsuna commented Nov 11, 2011

I can still reproduce with v 2.5.1

I also run into this one:

java.lang.IllegalStateException: Timer already cancelled.
        at java.util.Timer.sched(Timer.java:354) 
        at java.util.Timer.schedule(Timer.java:184) 
        at com.twitter.util.JavaTimer.schedule(Timer.scala:121) 
        at com.twitter.util.Future$$anonfun$within$1.apply(Future.scala:265) 
        at com.twitter.util.Future$$anonfun$within$1.apply(Future.scala:264) 
        at com.twitter.util.Future$.makePromise(Future.scala:174) 
        at com.twitter.util.Future.within(Future.scala:264) 
        at com.twitter.querulous.async.BlockingDatabaseWrapper.checkoutConnection(BlockingDatabaseWrapper.scala:69) 
        at com.twitter.querulous.async.BlockingDatabaseWrapper.withConnection(BlockingDatabaseWrapper.scala:47) 
        at com.twitter.querulous.async.StandardAsyncQueryEvaluator.withTransaction(StandardAsyncQueryEvaluator.scala:77) 
        at com.twitter.querulous.async.StandardAsyncQueryEvaluator.selectOne(StandardAsyncQueryEvaluator.scala:35) 
        at com.twitter.querulous.async.AsyncQueryEvaluator$class.selectOne(AsyncQueryEvaluator.scala:100) 
        at com.twitter.querulous.async.StandardAsyncQueryEvaluator.selectOne(StandardAsyncQueryEvaluator.scala:28) 
        at my.package.MySQL.selectOne(mysql.scala:51) 
@tsuna
tsuna commented Nov 11, 2011

I added some print statements in ApachePoolingDatabase in open() and close(), they're both called. According to the dbcp javadoc this is the right thing to do as close is supposed to return the connection to the pool.

I think either Querulous isn't using Apache's DBCP API properly, or there's a problem in Apache's code. All these layers of factories are giving me a nausea, I think I'll look at using another simpler connection pooling library or write my own.

@stevej
stevej commented Dec 6, 2011

This has been fixed, I believe, in the most recent versions. I ran into this issue a few weeks ago and fixed a race condition in FuturePool.

@tsuna
tsuna commented Dec 6, 2011

Can you give us the link to the fix?

@stevej
stevej commented Dec 6, 2011

This fix made JavaTimer more resilient
twitter/util@253606a

This fixed a race condition in FuturePool
twitter/util@7d6898a

@stevej
stevej commented Dec 6, 2011

If you can reproduce this with querulous 2.6.5+, please feel free to re-open this issue.

@tsuna
tsuna commented Dec 6, 2011

OK, but FWIW I do not believe my problem had to do with these bugs. I guess I was using Querulous incorrectly, but I still don't see how.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.