raftstore: make full compaction incremental, pause when load is high (#15995)

[ti-chi-bot]

This is an automated cherry-pick of #15995

Issue Number: ref #15271

Makes full compaction incremental, by range. Currently regions' ranges are used as increments.

Run a predicate ("load-check") function before starting full compaction and between each incremental
range.  If the function evaluates to false, pause with exponential backoff (up to a maximum duration)
until it evaluates to  true. 

If periodic full compaction is enabled, poll process CPU stats every 30 seconds to determine usage for the
"load-check"  function. If usage exceeds a certain threshold before full compaction starts, compaction will
 not be started, and if started, full compaction will be paused. This cpu usage is also exported as
``tikv_storage_process_stat_cpu_usage`` gauge metric.

Testing

• Manual tests.
• Unit test in compact.rs

Release note

Adds experimental support for periodic full compaction. Periodic full compaction ensures that SSTables are 
compactedat all levels, including the bottommost (L6.) This ensures compaction filters that purge deleted data
are able to run.  Since full compaction can impact live serving, periodic full compaction should only run during
times at which the load  is likely to low. As an additional precaution, periodic full compaction will not start (or if?
started, will be paused) if CPU usage exceeds a specified threshold.

Example tikv.toml configuration:
[raftstore]
periodic-full-compact-start-max-cpu = 0.33 
periodic-full-compact-start-times = ["03:00", "23:00"] 

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[ti-chi-bot]

This cherry pick PR is for a release branch and has not yet been approved by triage owners.
Adding the do-not-merge/cherry-pick-not-approved label.

To merge this cherry pick:

1. It must be approved by the approvers firstly.
2. AFTER it has been approved by approvers, please wait for the cherry-pick merging approval from triage owners.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.