Looks like adding checking transaction state dosn't help much. Deadlocks are disappeared, but the cancelled transaction remains associated with main thread and blocked new processing in this thread. I'm also registered this bug in jboss jira, may be they can help resolving it: https://issues.jboss.org/browse/JBTM-851
I'm afraid there isn't much we can do about this from Spring's side. We absolutely have to trigger a JTA rollback invocation in order to clean up transactional state that has been associated with the current thread. FWIW, any manual interaction with a JTA UserTransaction has to do the same.
So in order to be safe with any kind of JTA usage, not just with Spring's, such a check against an in-flight rollback has to happen within the JTA provider implementation itself.