ARTEMIS-1700 Fixed deadlock in paging state

[clebertsuconic]

This closes #1894

[clebertsuconic]

Please wait my ack before merging this.. I'm running the whole testsuite!

open for discussion only now

[franz1981]

if the latch is submitted from within the same thread that is executing the tasks, waiting won't be necessary, because it cannot be triggered

[clebertsuconic]

This won’t be used at all. It’s just the default implementation for tests. OrderedExecutor has an implementation.

[clebertsuconic]

it's ready to be merged! testsuite pass!

[shoukunhuai]

So it is a mistake to use global thread pool instead of io thread pool for page cursor.

But this does not fix our problem, as you can see

"Thread-274672 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@4e91d63f)" Id=274703 TIMED_WAITING on java.util.concurrent.CountDownLatch$Sync@5c416651
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.CountDownLatch$Sync@5c416651
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
	at org.apache.activemq.artemis.core.journal.impl.SimpleWaitIOCallback.waitCompletion(SimpleWaitIOCallback.java:73)
	at org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl.waitCompletion(OperationContextImpl.java:313)
	at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.waitOnOperations(AbstractJournalStorageManager.java:294)
	at org.apache.activemq.artemis.core.paging.cursor.impl.PageCursorProviderImpl.storeBookmark(PageCursorProviderImpl.java:539)
	at org.apache.activemq.artemis.core.paging.cursor.impl.PageCursorProviderImpl.cleanupComplete(PageCursorProviderImpl.java:431)
	at org.apache.activemq.artemis.core.paging.cursor.impl.PageCursorProviderImpl.cleanup(PageCursorProviderImpl.java:383)
	-  locked org.apache.activemq.artemis.core.paging.cursor.impl.PageCursorProviderImpl@4f8d6d9a
	at org.apache.activemq.artemis.core.paging.cursor.impl.PageCursorProviderImpl$1.run(PageCursorProviderImpl.java:291)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
	at org.apache.activemq.artemis.utils.actors.ProcessorBase$ExecutorTask.run(ProcessorBase.java:53)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

	Number of locked synchronizers = 2
	- java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@613d010
	- java.util.concurrent.ThreadPoolExecutor$Worker@4c03ae59

When exit paging state, we will store bookmark for each page subscription and wait until all callbacks done.
I believe this may happen even running in io thread as long as singleThreadExecutor in AbstractJournalStrorageManager use thread from global server thread pool.

[clebertsuconic]

You're right.. Man!!! you're good! :)

I will send another PR!

[clebertsuconic]

@shoukunhuai see #1904

[clebertsuconic]

@shoukunhuai please let me know if you still see any issues...

Testsuites have improved after this commit.