XA Transactions are running on multiple threads

[wkonecny]

We are trying to use the Hazelcast Resource Adapter with a Oracle Weblogic container. When a XA transaction started it gave an unexpected Exception.

After some research, it turns out that the transactions are running on multiple threads. However, Hazelcast explicitly checks for multiple threads and throws an Exception when this is the case. This causes a rollback to occur. What is the reasoning for not allowing multithreaded transactions?

com.hazelcast.transaction.impl.TransactionImpl.java:177

private void checkThread() {
    if (checkThreadAccess && threadId != null && threadId.longValue() != Thread.currentThread().getId()) {
        throw new IllegalStateException("Transaction cannot span multiple threads!");
    }
}

Additional Thead info

State	Thread no. Run 1	Thread no. Run 2
begin	15	15
prepare	53	64
commit	45	52

[mesutcelik]

@gurbuzali ,

any idea?

[wkonecny]

Has there been any progress regarding this issue?

[mdogan]

@mesutcelik, @gurbuzali; we need to have integration tests running on real xa transaction managers...

[toxjee]

Are there any updates on this subject? We'd really like to use the RA to handle our XA transactions,

[archiecobbs]

Seeing the same problem with Arjuna. Note the thread name here is "Transaction Reaper":

2014-10-14 10:35:36,470 [Transaction Reaper] WARN : ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff7f000001:90ea:543d32a3:191c in state  RUN
2014-10-14 10:35:36,471 [Transaction Reaper Worker 0] WARN : ARJUNA012095: Abort of action id 0:ffff7f000001:90ea:543d32a3:191c invoked while multiple threads active within it.
2014-10-14 10:35:36,472 [Transaction Reaper Worker 0] WARN : ARJUNA012108: CheckedAction::check - atomic action 0:ffff7f000001:90ea:543d32a3:191c aborting with 1 threads active!
2014-10-14 10:35:36,476 [Transaction Reaper Worker 0] WARN : ARJUNA016045: attempted rollback of < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff7f000001:90ea:543d32a3:191c, node_name=1, branch_uid=0:ffff7f000001:90ea:543d32a3:191d, subordinatenodename=null, eis_name=0 > (XAResourceImpl{txdId=1fc5799d-a89d-4543-816a-a856b3716d65, transactionTimeoutSeconds=0}) failed with exception code -
java.lang.IllegalStateException: Transaction cannot span multiple threads!
    at com.hazelcast.transaction.impl.TransactionImpl.checkThread(TransactionImpl.java:179)
    at com.hazelcast.transaction.impl.TransactionImpl.rollback(TransactionImpl.java:330)
    at com.hazelcast.transaction.impl.XAResourceImpl.rollback(XAResourceImpl.java:148)
    at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelAbort(XAResourceRecord.java:368)
    at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:2980)
    at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:2959)
    at com.arjuna.ats.arjuna.coordinator.BasicAction.Abort(BasicAction.java:1666)
    at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.cancel(TwoPhaseCoordinator.java:127)
    at com.arjuna.ats.arjuna.AtomicAction.cancel(AtomicAction.java:215)
    at com.arjuna.ats.arjuna.coordinator.TransactionReaper.doCancellations(TransactionReaper.java:381)
    at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:78)
2014-10-14 10:35:36,480 [Transaction Reaper Worker 0] WARN : ARJUNA012091: Top-level abort of action 0:ffff7f000001:90ea:543d32a3:191c received TwoPhaseOutcome.FINISH_ERROR from com.arjuna.ats.arjuna.coordinator.AbstractRecord
2014-10-14 10:35:36,480 [Transaction Reaper Worker 0] WARN : ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffff7f000001:90ea:543d32a3:191c

What's worse, once this happens, the original thread is rendered incapable of ever opening another Hazelcast transaction because a ThreadLocal never got cleared. Instead every subsequent attempt to open a transaction in that thread results in:

java.lang.IllegalStateException: Nested transactions are not allowed!
    at com.hazelcast.transaction.impl.TransactionImpl.begin(TransactionImpl.java:188)
    at com.hazelcast.transaction.impl.XAResourceImpl.start(XAResourceImpl.java:63)
    at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:662)
    at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:423)

So Hazelcast XA support is not really usable ... unless your code never, ever rolls back a transaction :)

[markathomas]

Does anyone know why Hazelcast checks the thread? Perhaps one of the Hazelcast devs can shine some light on the matter? I would think multiple thread access would be common place in a JTA environment.

[archiecobbs]

I would guess they are trying to detect and prevent nested (i.e., reentrant) transactions.

[bwzhang2011]

please take a look at #451. hope hz team could solve transaction completely not long in the future.

[djvdorp]

It would be really great if this gets fixed, enterprises use multithreaded XA Transactions a lot!

[juanavelez]

Any updates? We are really interested in using HZ within our JBoss EAP transactions but this is a no-no if it is not fixed.

[jerrinot]

@gurbuzali This one is gonna be fixed in 3.5. Is it correct?

[gurbuzali]

should be fixed in EA

[jerrinot]

This is fixed in the current master and it'll be included in Hazelcast 3.5.