Add default message handling retry count for state transition messages.

[jiajunwang]

Issues

• My PR addresses the following Helix issues and references them in the PR description:

Resolves #1513

Description

• Here are some details about my PR, including screenshots of any UI changes:

The retry count will be counted when message handler creation fails or state transition times out.

In addition, this PR also improves several ambiguous behaviors that may block a normal retry.

1. The cache of known messages list in HelixTaskExecutor may not be updated correctly if a message is going to be retried.
2. The retry count is not strictly followed in some conditions.
3. The to-be-retried message is not automatically read when there are no other new message changes. In this PR we change this behavior and ensure the retry will be done even no other message created.

Finally, improve the tests to cover the new changes.

Tests

• The following tests are written for this issue:

TestStateTransitionAppFailureHandling

• The following is the result of the "mvn test" command on the appropriate module:

[ERROR] Failures:
[ERROR] TestTaskRebalancerFailover.test » ThreadTimeout Method org.testng.internal.Tes...
[INFO]
[ERROR] Tests run: 1247, Failures: 1, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:36 h
[INFO] Finished at: 2020-11-06T00:46:51-08:00
[INFO] ------------------------------------------------------------------------

Rerun the test and it will pass. But it is not stable due to issue #1512

Documentation (Optional)

• In case of new functionality, my PR adds documentation in the following wiki page:

(Link the GitHub wiki you added)

Commits

• My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Code Quality

• My diff has been formatted using helix-style.xml
  (helix-style-intellij.xml if IntelliJ IDE is used)

[jiajunwang]

Rebased to the current master and retested.

[INFO] Results:
[INFO]
[INFO] Tests run: 1247, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:29 h
[INFO] Finished at: 2020-11-10T14:54:05-08:00
[INFO] ------------------------------------------------------------------------

[jiajunwang]

Thank @dasahcc for the approval. I will merge the code soon.