HostAvailabilityListener with QueryBatcher results in incorrect behavior

[srinathgit]

A. The test used here is the same as reference in #568 . It queries all documents in collection "Replace Snapshot", applies transform on them and then finally deletes the document in a single QueryBatcher instance.
B. This test was run on a 3 node cluster (rh7v-intel64-90-test-4/5/6.marklogic.com) with a forest (ApplyTransform1,2,3) on each of the node associated with a db.
C. The forest on node rh7v-intel64-90-test-6.marklogic.com has been configured for shared disk failover to failover host rh7v-intel64-90-test-4.marklogic.com.
D. Now when 'batcher ' QB job is getting executed,node rh7v-intel64-90-test-6.marklogic.com is stopped.
E. After 30 seconds, the forest fails over to rh7v-intel64-90-test-4.marklogic.com and db once again becomes available
F. But from the logs, it can be seen that the document with uri "/local/snapshot-0" has not been deleted from the db (it was confirmed that the document was available using QConsole as well).

Client log

Test:

java.lang.AssertionError
	at org.junit.Assert.fail(Assert.java:86)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.junit.Assert.assertFalse(Assert.java:64)
	at org.junit.Assert.assertFalse(Assert.java:74)
	at com.marklogic.client.datamovement.functionaltests.ApplyTransformTest.jsMasstransformReplaceDelete(ApplyTransformTest.java:669)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

[srinathgit]

I ran the same test on a 3 node cluster (rh7v-intel64-90-test-19/20/21.marklogic.com) with a forest (WriteHostBatcher-1,2,3) on each of the node associated with a db. I now see that the reason why 6 documents weren't transformed/ deleted is because the server was stopped and there are no retries for either apply transform or delete listeners and hence they are marked as failures in client log. This seems to be working as expected. Can you please confirm my observation ?

clientLog

[srinathgit]

This issue is not seen anymore