Fix Periodic rebalancer Timer leak

[xyuanlu]

Issues

• My PR addresses the following Helix issues and references them in the PR description:
  Race condition cause Timer leak in GenericHelixController #1453 Race condition cause Timer leak in GenericHelixController

Description

• Here are some details about my PR, including screenshots of any UI changes:
  We found a periodic rebalance's Timer leakage issue during log analysis. In current startPeriodRebalance, two thread may interference with each other. This may result in one timer got canceled twice, two timers are created with one timer leaked.

  void startPeriodRebalance(long period, HelixManager manager) {
    if (period != _timerPeriod) {
      logger.info("Controller starting periodical rebalance timer at period " + period);
      if (_periodicalRebalanceTimer != null) {
        _periodicalRebalanceTimer.cancel();                  <<<------ 
      } 
      _periodicalRebalanceTimer =
          new Timer("GenericHelixController_" + _clusterName + "_periodical_Timer", true);     <<<-----
     ......
  }

This PR changes Timer to use a SingleThreadScheduledExecutor and adds a synchronized block in start/stop PeriodRebalance.

Tests

• The following tests are written for this issue:

NA

• The following is the result of the "mvn test" command on the appropriate module:

[ERROR] Failures: 
[ERROR]   TestRoutingTableProviderPeriodicRefresh.testPeriodicRefresh:214 expected:<4> but was:<3>
[INFO]
[ERROR] Tests run: 1234, Failures: 1, Errors: 0, Skipped: 1
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:23 h

Commits

• My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation (Optional)

• In case of new functionality, my PR adds documentation in the following wiki page:

(Link the GitHub wiki you added)

Code Quality

• My diff has been formatted using helix-style.xml
  (helix-style-intellij.xml if IntelliJ IDE is used)

[kaisun2000]

One more thing I forget to mention before is this:

The current leaking is caused that two events are push to TaskRebalance thread and HelixRebalance thread at the same time.
For this kind of changing the config time period change, in fact, we don't need to put two events to two different thread running task or helix separately.

All the above is under assumption that periodic rebalance triggering thread (previous timer, now the is shared by task and now _periodicalRebalanceExecutor. Is there a possibility later that we will use two different (timer)?

[jiajunwang]

LGTM

Please ensure the remaining comments are addressed.

[xyuanlu]

My PR is ready to be merged in. Approved by @jiajunwang

Final commit message:
Fix Periodic rebalancer Timer leak
In current startPeriodRebalance, two thread may interference with each other. This may result in one timer got canceled twice, two timers are created with one timer leaked. This PR changes Timer to use a SingleThreadScheduledExecutor and adds a synchronized block in start/stop PeriodRebalance.