Deadlock in multi-threaded environment [JDBC435]

[firebird-automations]

Submitted by: @reevespaul

Is duplicated by JDBC434

Attachments:
deadlock.jpeg

When closing multiple threaded connections the driver hangs. The problem seems to be related to the test for mc == null in isClosed()

I tried a while ago with 2.2.3 and most recently with the code in the v3.0.0 snapshot. Neither work.

The last version that works for me is 2.1.6.

Commits: 78d882c a085ae0 7cd423e 4d7f118

[firebird-automations]

Commented by: @reevespaul

Hopefully this screen shot from netbeans will give a better idea of what is happening. If necessary I can provide the app and the database.

[firebird-automations]

Modified by: @reevespaul

Attachment: deadlock.jpeg [ 12963 ]

[firebird-automations]

Modified by: @mrotteveel

Link: This issue is duplicated by JDBC434 [ JDBC434 ]

[firebird-automations]

Commented by: @mrotteveel

I will investigate this. Having access to the app and database would be very helpful.

As a side note, although JDBC asks that drivers are threadsafe, in general you shouldn't use the same connection from multiple threads simultaneously.

[firebird-automations]

Commented by: @mrotteveel

The problem is not with the null check in isClosed. It is with the fact that the order that executeBatch first acquires a lock on the statement, and later on on the connection, while connection close first acquires a lock on the connection, and then (when closing statements in response to the connection close) on the statement.

[firebird-automations]

Commented by: @reevespaul

Just to comment on your side note - no, in the application each thread creates its own connection. With 2.16 I've had up to 100 concurrent threads working without problem.

But good to hear that you may have identified the problem. I can test the fix easily enough when it is ready.

[firebird-automations]

Commented by: @mrotteveel

The screenshot seems to indicate that the connection and statements are shared: Thread 4 and Thread 6 in the screenshot are deadlocked on the monitors of the same connection and statement.

[firebird-automations]

Commented by: @reevespaul

But where does that monitor come from?
It is certainly not something in the application code, as far as I am aware. I grepped 'monitor' and found nothing.

Is it something in the JVM? Jaybird?

[firebird-automations]

Commented by: @mrotteveel

Monitor is the technical term for what you lock when code does synchronized(object) { .... } (or use a synchronized method) It is an integral part of the object being locked on (so it is a fundamental part of Java).

In this case, it seems to be one and the same connection used by more than one thread.

[firebird-automations]

Commented by: @reevespaul

Which poses an interesting question. Is the synchronization problem in my code or is it something in Jaybird?

I just checked with Jaybird 2.1.6 and there is no deadlock. On the other hand I can reproduce this consistently with Jaybird 3.0.0 snapshot running in a debug session in netbeans.

[firebird-automations]

Commented by: @mrotteveel

Paul, could you send me the database + the code (or a download link to it)?

[firebird-automations]

Modified by: @mrotteveel

Fix Version: Jaybird 3.0 [ 10440 ]

[firebird-automations]

Commented by: @mrotteveel

I decided to make the locking a little bit more coarse, with a single lock object for all related objects (eg the statement lock object is the same as the connection lock object, etc). I'm first doing this in the http://org.firebirdsql.gds.ng.* objects, and will check if I will also need to do something similar with the org.firebirdsql.jdbc.* objects. As connections etc should not really be used from multiple threads at the same time, having these coarser locks should not impact performance for most use cases. It might have an impact on use case where a single connections is shared by multiple threads.

It would still be very helpful to have the code to test with.

[firebird-automations]

Commented by: @mrotteveel

In Jaybird 3, I applied a coarser synchronization with - where possible - a single lock object per connection.

This should prevent the potential of deadlock as described above, which may occur because in some situations the locks are obtained in order connection->statement, and in others statement->connection (etc).

I've not yet decided if I'm going to backport this to Jaybird 2.2 or not.

[firebird-automations]

Modified by: @mrotteveel

Fix Version: Jaybird 2.2.11 [ 10751 ]