Reconsider bg_worker runtime size and dedicated heartbeat tasks

[sunng87]

What type of enhancement is this?

Refactor

What does the enhancement do?

In a resource restrict environment, we saw bg_worker tasks (flush) utilize all the CPU resource that:

1. heartbeat tasks are blocked, cause the node into readonly state Region 265098266411008(61723, 0) is in ReadOnly state, expect: Writable #4122
2. k8s http liveness probe failed with timeout on simple tasks like curling /health

The proposal is to:

1. Change default size of bg_worker to half the cpu number, so in worst case it won't take all cpus for other tasks.
2. Move critial tasks like heartbeat to dedicated runtime so it won't be affected when bg_workers filled up with cpu-intensive tasks

Implementation challenges

No response