Assignment metadata missing for the WAGED rebalancer in stress test.

[jiajunwang]

Please see the attachment, the baseline node is empty. I observed this multiple times in the stress test.

I suspect this is about the GC thread.

1. ZkBucketDataAccessor.GC_THREAD should be created as a single thread pool instead of creating by newScheduledThreadPool() which could have more threads.
2. To ensure the order of the clean tasks, we'd better leverage a queue instead of relying on the TTL.

[jiajunwang]

@narendly Could you please take a look at this issue? We will need to fix it before we can put Waged rebalancer into large scale usage.

[NealSun96]

@jiajunwang Taking over this issue.

[jiajunwang]

There is another concurrent modification issue found.

1. Regular GC task fails due to concurrent list modification and the stale versions are not removed at all.
2. If, by coincident, there is newer version in the list other then the current version, then because of the modification of the list inside the loop, the final element (the newer version) won't be filtered but being left in the to-be-removed list. Then the GC task removes the most recent version. For example,
   a) Input, current version "2"
   b) Children = [1, 2, 3]
   c) The task avoids checking "2", so the list for loop is: [1, 3]
   d) When check "1", it is removed from the list. So the list becomes [3]. Then the loop ends, because the first item has already been looped from the for iteration perspective.
   e) The version to be removed is "3"!