[GCS]Cherry pick heartbeat function into another thread

[WangTaoTheTonic]

Why are these changes needed?

Related issue number

In normal cases raylet main thread doesn't do too much heavy load, but sometime it could be blocked by some lock operation( we found servral times).
Heartbeat is a light but import action for thread, as it is depended by gcs node failure detecting. To avoid raylet being treated as lost by misunderstand, we'd like it sending heartbeat in a different thread from the main one.

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[raulchen]

LGTM. One minor comment is that I prefer adding a separate HeartbeatSender class. Because NodeManager is already too huge.

[rkooo567]

Hey do we still need this @wuisawesome with your change? Can you confirm?

[wuisawesome]

I'm in favor of this since it should help prevent issues like #14264