raftstore: make full compaction incremental, pause when load is high

[afeinberg]

Issue Number: ref #15271

Makes full compaction incremental, by range. Currently regions' ranges are used as increments.

Run a predicate ("load-check") function before starting full compaction and between each incremental
range.  If the function evaluates to false, pause with exponential backoff (up to a maximum duration)
until it evaluates to  true. 

If periodic full compaction is enabled, poll process CPU stats every 30 seconds to determine usage for the
"load-check"  function. If usage exceeds a certain threshold before full compaction starts, compaction will
 not be started, and if started, full compaction will be paused. This cpu usage is also exported as
``tikv_storage_process_stat_cpu_usage`` gauge metric.

Testing

• Manual tests.
• Unit test in compact.rs

Release note

Adds experimental support for periodic full compaction. Periodic full compaction ensures that SSTables are 
compactedat all levels, including the bottommost (L6.) This ensures compaction filters that purge deleted data
are able to run.  Since full compaction can impact live serving, periodic full compaction should only run during
times at which the load  is likely to low. As an additional precaution, periodic full compaction will not start (or if?
started, will be paused) if CPU usage exceeds a specified threshold.

Example tikv.toml configuration:
[raftstore]
periodic-full-compact-start-max-cpu = 0.33 
periodic-full-compact-start-times = ["03:00", "23:00"] 

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• LykxSassinator
• v01dstar

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[afeinberg]

/cc @tonyxuqqi @v01dstar

[afeinberg]

/test

[afeinberg]

/cc @Connor1996

[v01dstar]

I thought we were trying to get cpu usage "over time" instead of "point in time"?

[afeinberg]

after discussion with @tonyxuqqi we decided to decide on cpu usage at every 10 minute interval. so we are doing over time, but that time is a 10 minute interval. (we call it every 10 minutes, and it returns the cpu usage as defined by cpu time - cpu time last time process_stat.cpu_usage() is gotten / time - it was last called.)

[v01dstar]

LGTM

[afeinberg]

/assign @tonyxuqqi

[tonyxuqqi]

/merge

[ti-chi-bot]

@tonyxuqqi: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: f6e8ebd

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-7.5: #17034.