Fix flaky tests

[narendly]

[ERROR] TestRebalancePipeline.testMsgTriggeredRebalance:195 expected:<1> but was:<0>
[ERROR] TestEnableCompression.testEnableCompressionResource:117 expected: but was:

[jiajunwang]

Let me take a look at the testMsgTriggeredRebalance failure.

[jiajunwang]

Adding 2 unstable tests that happen in the other test runs.

TestControllerLeadershipChange.testMissingTopStateDurationMonitoring:262 expected: but was:
org.apache.helix.integration.rebalancer.CrushRebalancers.TestCrushAutoRebalanceNonRack.testLackEnoughInstances(org.apache.helix.integration.rebalancer.CrushRebalancers.TestCrushAutoRebalanceNonRack)
Run 1: PASS
Run 2: TestCrushAutoRebalanceNonRack.testLackEnoughInstances:250 » ZkClient Failed to...

[huizhilu]

Results :

Failed tests:
  TestBasicSpectator.TestSpectator:58 expected:<true> but was:<false>
org.apache.helix.integration.rebalancer.CrushRebalancers.TestCrushAutoRebalanceTopoplogyAwareDisabled.testLackEnoughInstances(org.apache.helix.integration.rebalancer.CrushRebalancers.TestCrushAutoRebalanceTopoplogyAwareDisabled)
  Run 1: PASS
  Run 2: TestCrushAutoRebalanceTopoplogyAwareDisabled.testLackEnoughInstances:86->TestCrushAutoRebalanceNonRack.testLackEnoughInstances:250 » ZkClient

  TestWorkflowTermination.testWorkflowJobFail:251->verifyWorkflowCleanup:257 expected:<true> but was:<false>
  TestWorkflowTermination.testWorkflowRunningTimeout:131->verifyWorkflowCleanup:257 expected:<true> but was:<false>

Tests run: 1111, Failures: 4, Errors: 0, Skipped: 1

[huizhilu]

[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] org.apache.helix.integration.rebalancer.CrushRebalancers.TestCrushAutoRebalanceNonRack.test(org.apache.helix.integration.rebalancer.CrushRebalancers.TestCrushAutoRebalanceNonRack)
[INFO]   Run 1: PASS
[ERROR]   Run 2: TestCrushAutoRebalanceNonRack.test:165->validateIsolation:313 expected:<3> but was:<4>
[INFO]
[INFO]
[ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 3

[huizhilu]

Results :

Failed tests:
TestBasicSpectator.TestSpectator:58 expected: but was:
org.apache.helix.integration.rebalancer.CrushRebalancers.TestCrushAutoRebalanceTopoplogyAwareDisabled.testLackEnoughInstances(org.apache.helix.integration.rebalancer.CrushRebalancers.TestCrushAutoRebalanceTopoplogyAwareDisabled)
Run 1: PASS
Run 2: TestCrushAutoRebalanceTopoplogyAwareDisabled.testLackEnoughInstances:86->TestCrushAutoRebalanceNonRack.testLackEnoughInstances:250 » ZkClient

Tests run: 1111, Failures: 4, Errors: 0, Skipped: 1

1. In ZkClient.deleteRecursively(), getChildren(path) for

/CLUSTER_TestCrushAutoRebalanceNonRack/INSTANCES/localhost_12921/MESSAGES is
[c024282d-b461-47ae-994a-3e951b33bda8, 81c9d998-a4f7-463d-b7d2-444851ac3b94, 8b0aae78-4f4a-488a-8b57-60c1f5cba2a0, e51f3138-0d7b-44ff-96d9-9b199dedd945, 6
803ce21-818f-404f-84d8-c39d60e28c2d, c8a22399-de1b-4d7e-8266-6712e16dd0bc, 7b6428ad-1da2-40b6-bc00-44091723df68, ed661695-0c32-42cf-bf76-08da4914c40a, 257f61e5-7238-4a
fa-867b-003275d63708, 3c66083b-48f6-48f5-b59d-33b94f246cc8, 35c0f0af-2bca-40bd-b23f-f0c417b43504, 3cb84f5f-d121-4492-a5a5-230c39c8b621, c5f9e7db-fc9f-4f33-9770-7fc86ac
fe0b2, 90cfa4a1-8632-49c7-abc0-2e4ac14d5870, baf6698f-e9a5-433e-a9ef-02af0218992e, 253f5bd6-d703-4a37-8aa3-1b2f688c5d4e, b97e8cca-a498-47b7-a331-21423a687c48, 0f43e755
-fa3c-444f-b588-4a93ac10b05f, cd9b3160-9b40-422c-828d-c2890f6d6e5e]

Later after all the previously fetch children are deleted and the path is being deleted, the path has children again:

2020-04-13 01:01:32,393 [main] ERROR org.apache.helix.zookeeper.zkclient.ZkClient - children: [99db6e1a-26dd-4abb-bc67-420f2ce17333, c563c2d0-d20a-46ed-823a-69bd5d1b16ad, ac8b5651-03a1-4dbf-a659-16a45381e9a3, 2d93d9a7-72e5-4381-a432-f88cc28b73b3, 04b43472-d372-4841-a006-f0faa180f829, fae92659-4e3a-46e6-8ec0-349811ead778, 444f8824-5cc5-4e51-ab42-5ce0b917364a, ed77a721-f021-432a-ab4b-a5f2a7d34a5c, b8cd4995-8119-4675-b1e6-09b3e5ff7c0f, 27550b68-0af8-4cad-8fbc-704e90e39c3d, 8e576677-8b3d-459e-9e94-1dabc33f9a46, 5c033c68-ec61-41b0-8e5e-bb3a8dd750a4, 277c3c23-46f1-4683-ac58-2bf1ef045cea, dc263f96-ec5a-47a2-92e3-ebc1776e99c2, c37655ab-3ad2-4797-ac99-a4fbd1775e65, 3e236227-87c4-47db-9d2c-0f4340afd87c, 3e06b4bc-3485-4927-81a7-84f95ae93597, 8205b6a3-f7e5-4abf-8e30-404a4d24fe9a, 4ee60a6c-fd70-4240-9c27-5d801a263b14, e4a22424-3216-4e98-a9f0-fe25338c9260, c0d2ccf7-bed6-489d-856e-f4e85262d597, 2c72f45e-d4cb-4c24-adf1-fd8183912207, e01fb704-ffd4-41a8-95e1-9eed0590a770, a5e51478-ae7a-48c6-9e42-8ce1269cd288]

From the log, messages are sent to the path during deleteRecursively().

2020-04-13 01:01:31,732 [HelixController-pipeline-default-CLUSTER_TestCrushAutoRebalanceNonRack-(ff4d92c0_DEFAULT)] INFO  org.apache.helix.controller.stages.MessageDis
patchStage - Event ff4d92c0_DEFAULT : Sending Message 99db6e1a-26dd-4abb-bc67-420f2ce17333 to localhost_12921 transit Test-DB-CrushRebalanceStrategy-0.Test-DB-CrushReb
alanceStrategy-0_18|[] from:OFFLINE to:DROPPED, relayMessages: 0

It seems messages are not processed completely while dropping the instance from cluster.

2. One more thing to notice is, ZkHelixAdmin.dropInstance() has a retry. But in the log retry is not executed. The root cause is ZkClient.deleteRecursively() changes the throwing exception from HelixException to ZkClientException but ZkHelixAdmin.dropInstance() is still catching HelixException to retry. So ZkClientException is not caught and no retry to delete the path.

[huizhilu]

-------------------------------------------------------------------------------
Test set: TestSuite
-------------------------------------------------------------------------------
Tests run: 1119, Failures: 3, Errors: 0, Skipped: 1, Time elapsed: 4,883.15 s <<< FAILURE! - in TestSuite
testFixedTargetTaskAndDisabledRebalanceAndNodeAdded(org.apache.helix.integration.task.TestRebalanceRunningTask)  Time elapsed: 11.404 s  <<< FAILURE!
java.lang.AssertionError: expected:<true> but was:<false>
        at org.apache.helix.integration.task.TestRebalanceRunningTask.testFixedTargetTaskAndDisabledRebalanceAndNodeAdded(TestRebalanceRunningTask.java:265)

testDeletingRecurrentQueueWithHistory(org.apache.helix.integration.task.TestRecurringJobQueue)  Time elapsed: 121.864 s  <<< FAILURE!
java.lang.AssertionError: expected:<true> but was:<false>
        at org.apache.helix.integration.task.TestRecurringJobQueue.testDeletingRecurrentQueueWithHistory(TestRecurringJobQueue.java:296)

testJobQueueAutoCleanUp(org.apache.helix.integration.task.TestJobQueueCleanUp)  Time elapsed: 300.002 s  <<< FAILURE!
org.testng.internal.thread.ThreadTimeoutException: Method org.testng.internal.TestNGMethod.testJobQueueAutoCleanUp() didn't finish within the time-out 300000

[huizhilu]

TestZkMetadataStoreDirectory failure.

[ERROR] Tests run: 11, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 18.857 s <<< FAILURE! - in org.apache.helix.rest.metadatastore.TestZkMetadataStoreDirectory
[ERROR] testSetNamespaceRoutingData(org.apache.helix.rest.metadatastore.TestZkMetadataStoreDirectory)  Time elapsed: 0.027 s  <<< FAILURE!
java.util.NoSuchElementException: Namespace namespace_0 does not exist!
	at org.apache.helix.rest.metadatastore.TestZkMetadataStoreDirectory.testSetNamespaceRoutingData(TestZkMetadataStoreDirectory.java:175)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   TestZkMetadataStoreDirectory.testSetNamespaceRoutingData:175 » NoSuchElement N...
[INFO]
[ERROR] Tests run: 11, Failures: 1, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------




[ERROR] Tests run: 11, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 18.897 s <<< FAILURE! - in org.apache.helix.rest.metadatastore.TestZkMetadataStoreDirectory
[ERROR] testDataDeletionCallback(org.apache.helix.rest.metadatastore.TestZkMetadataStoreDirectory)  Time elapsed: 0.029 s  <<< FAILURE!
java.lang.IllegalArgumentException: Provided path is not a valid Zookeeper path: anyKey
	at org.apache.helix.rest.metadatastore.TestZkMetadataStoreDirectory.lambda$testDataDeletionCallback$10(TestZkMetadataStoreDirectory.java:340)
	at org.apache.helix.rest.metadatastore.TestZkMetadataStoreDirectory.testDataDeletionCallback(TestZkMetadataStoreDirectory.java:337)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   TestZkMetadataStoreDirectory.testDataDeletionCallback:337->lambda$testDataDeletionCallback$10:340 » IllegalArgument
[INFO]
[ERROR] Tests run: 11, Failures: 1, Errors: 0, Skipped: 0

[jiajunwang]

Close test unstable tickets since we have an automatic tracking mechanism #1757 now for tracking the most recent test issues.