Deadlock when loading DataType classes

[tstack]

We're seeing a deadlock when two different threads load different DataType classes at the same time. It looks like loading one DataType class causes the others to be loaded as well during class initialization. But, this can cause a deadlock when multiple threads are trying to load these DataType classes because they each hold a lock on the class and end up waiting for the others to finish.

The following code reproduces the deadlock for me:

import java.io.*;
import java.net.*;

class jooqlock {
    public static void main(String[] args) throws Exception {
        for (int i = 0; i < 1; i++) {
            doit();
        }
    }

    private static void doit() throws Exception {
        final ClassLoader loader = new URLClassLoader(new URL[] { new File("jooq-3.4.4.jar").toURI().toURL() });

        Runnable oload = new Runnable() {
                public void run() {
                    try {
                        Class.forName("org.jooq.util.sqlite.SQLiteDataType", true, loader);
                    } catch (Throwable th) {
                        th.printStackTrace();
                    }
                }
            };
        Runnable sload = new Runnable() {
                public void run() {
                    try {
                        Class.forName("org.jooq.util.postgres.PostgresDataType", true, loader);
                    } catch (Throwable th) {
                        th.printStackTrace();
                    }
                }
            };

        Thread th1 = new Thread(oload);
        Thread th2 = new Thread(sload);
        th1.start();
        th2.start();
        th1.join();
        th2.join();
    }
}

Here is a stack dump from after the deadlock happens:

Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode):

"Thread-1" prio=5 tid=0x00007fd119841800 nid=0x5303 in Object.wait() [0x000000011b619000]
   java.lang.Thread.State: RUNNABLE
    at org.jooq.util.postgres.PostgresDataType.<clinit>(PostgresDataType.java:76)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:270)
    at jooqlock$2.run(jooqlock.java:27)
    at java.lang.Thread.run(Thread.java:745)

"Thread-0" prio=5 tid=0x00007fd119841000 nid=0x5103 in Object.wait() [0x000000011b515000]
   java.lang.Thread.State: RUNNABLE
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:190)
    at org.jooq.impl.SQLDataType.<clinit>(SQLDataType.java:341)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:190)
    at org.jooq.impl.DefaultDataType.<clinit>(DefaultDataType.java:219)
    at org.jooq.util.sqlite.SQLiteDataType.<clinit>(SQLiteDataType.java:71)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:270)
    at jooqlock$1.run(jooqlock.java:18)
    at java.lang.Thread.run(Thread.java:745)

[lukaseder]

Thank you very much for letting us know. Yes, this can indeed happen, specifically because of the static block in org.jooq.impl.SQLDataType, which we implemented to prevent that sort of issue:

    static {
        // Load all dialect-specific data types
        // TODO [#650] Make this more reliable using a data type registry

        try {
            Class.forName(CUBRIDDataType.class.getName());
            Class.forName(DerbyDataType.class.getName());
            Class.forName(FirebirdDataType.class.getName());
            Class.forName(H2DataType.class.getName());
            Class.forName(HSQLDBDataType.class.getName());
            Class.forName(MariaDBDataType.class.getName());
            Class.forName(MySQLDataType.class.getName());
            Class.forName(PostgresDataType.class.getName());
            Class.forName(SQLiteDataType.class.getName());
        } catch (Exception ignore) {}
    }

Before we look into fixing this, could you tell me why this might happen in your specific use-case? Do you have generated classes that refer to the dialect-specific data types early on?

[tstack]

We have some helper classes that are initialized early on and refer to the DataType classes, we're not using generated classes. We also use multiple database types at the same time in different threads in the jvm. Our fix is to load jooq separately for each thread since they already had their own class loaders.

I imagine our situation is not very common, but we wanted to let you know.

[lukaseder]

I imagine our situation is not very common, but we wanted to let you know.

Yes, it's not very common, but I guess we were taking bets when we implemented the above, and Murphy's law (as always) struck. In fact, we knew that it would strike (see #2322), but we couldn't think of a scenario to produce that issue...

Thanks for letting us know about this. We'll think of a fix that'll make initialisation more robust.

[lukaseder]

Related #2322

[lukaseder]

I've found two possible solutions:

1. Use reflection to check whether the class is already loaded. This is more of a quick fix. Example here: http://stackoverflow.com/a/482909/521799
2. Create a new class that initialises all dialects, and dialect-specific data types, and let PostgresDataType.* simply be references to the other class's data types.

The latter would be much more robust in the sense of #2322. But I think it won't make it into 3.5.0 anymore. It'll need some more thinking.