Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make database connection errors more explicit #1029

Merged
merged 10 commits into from Jun 24, 2015

Conversation

rnsmith
Copy link
Contributor

@rnsmith rnsmith commented Jun 3, 2015

Attempt to make errors caused by database connection failure more obvious by throwing/handling a specific DatabaseConnectionException. Hopefully fixes #919 and #912 but hard to test periodic errors.

@julie-sullivan
Copy link
Member

@rnsmith How would you recommend I test this?

@rnsmith
Copy link
Contributor Author

rnsmith commented Jun 3, 2015

It's quite hard to test reliably. If you set maxConnections for db.production and db.common-tgt-items in default.intermine.intergrate.properties to low numbers (1-3) and run a build you should see various connection errors. You will need to vary the numbers to see connection errors at different points in the build.

All exceptions should clearly tell you the database has run out of connections. If they don't then more changes are needed.

@julie-sullivan
Copy link
Member

build-db for FlyMine didn't fail as expected:

-do-build-db:
 [build-db] Our new url -> jdbc:postgresql://localhost/flymine
 [build-db] Our new url -> jdbc:postgresql://localhost/flymine

BUILD FAILED
/local-homes/julie/git/intermine/imbuild/objectstore.xml:52: Error creating indexes

@julie-sullivan
Copy link
Member

Running an individual source didn't give me the error message either.

BUILD FAILED
/local-homes/julie/git/intermine/imbuild/integrate.xml:54: The following error occurred while executing this line:
/local-homes/julie/git/intermine/imbuild/source.xml:145: java.lang.Error: Failed to get database connection when initiating PrecomputedTableManager
    at org.intermine.objectstore.intermine.ObjectStoreInterMineImpl.databaseAltered(ObjectStoreInterMineImpl.java:1724)
    at org.intermine.objectstore.intermine.ObjectStoreWriterInterMineImpl.commitTransactionWithConnection(ObjectStoreWriterInterMineImpl.java:1569)
    at org.intermine.objectstore.intermine.ObjectStoreWriterInterMineImpl.commitTransaction(ObjectStoreWriterInterMineImpl.java:1547)
    at org.intermine.dataconversion.ObjectStoreItemWriter.close(ObjectStoreItemWriter.java:130)
    at org.intermine.task.FileConverterTask.execute(FileConverterTask.java:133)
    at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:292)
       ...
Caused by: org.intermine.sql.DatabaseConnectionException: Unable to open database connection: {platform=PostgreSQL, datasource.dataSourceClassName=org.postgresql.ds.PGSimpleDataSource, datasource.dataSourceName=db.common-tgt-items, datasource.maxConnections=1, datasource.databaseName=items, datasource.class=com.zaxxer.hikari.HikariDataSource, datasource.password=----, datasource.user=----, datasource.serverName=localhost, driver=org.postgresql.Driver} org.postgresql.Driver PostgreSQL
    at org.intermine.sql.Database.getConnection(Database.java:173)
    at org.intermine.sql.precompute.PrecomputedTableManager.<init>(PrecomputedTableManager.java:88)
    at org.intermine.sql.precompute.PrecomputedTableManager.getInstance(PrecomputedTableManager.java:130)
    at org.intermine.objectstore.intermine.ObjectStoreInterMineImpl.databaseAltered(ObjectStoreInterMineImpl.java:1721)
    ... 35 more
Caused by: java.sql.SQLException: Timeout of 30000ms encountered waiting for connection.
    at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:184)
    at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:84)
    at org.intermine.sql.Database.getConnection(Database.java:169)
    ... 38 more

@rnsmith
Copy link
Contributor Author

rnsmith commented Jun 17, 2015

It is giving the correct error message in both cases, I've improved the message.

It is possible to get the numbers of active and idle connections from Hikari, I played around with it for a while and couldn't get it working nicely with instances of our Database.java class. Would be good to add in future, see:

http://stackoverflow.com/questions/24187891/get-active-connection-on-hikaridatasource
https://github.com/brettwooldridge/HikariCP/wiki/MBean-%28JMX%29-Monitoring-and-Management

@julie-sullivan
Copy link
Member

Here's the error message I get when trying to run build-db:

Caused by: org.intermine.sql.DatabaseConnectionException: Unable to open database connection (there may not be enough available connections): {platform=PostgreSQL, datasource.dataSourceClassName=org.postgresql.ds.PGSimpleDataSource, datasource.dataSourceName=db.production, datasource.maxConnections=1, datasource.databaseName=flymine, datasource.class=com.zaxxer.hikari.HikariDataSource, datasource.password=xxx, datasource.user=xxx, driver=org.postgresql.Driver, datasource.serverName=localhost} org.postgresql.Driver PostgreSQL
    at org.intermine.sql.Database.getConnection(Database.java

@julie-sullivan
Copy link
Member

And the same for the individual source in /integrate. Looks good @rnsmith! Thanks. Will merge in once 1.5.4 is finished.

julie-sullivan pushed a commit that referenced this pull request Jun 24, 2015
Make database connection errors more explicit
@julie-sullivan julie-sullivan merged commit c6381cf into intermine:beta Jun 24, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants