Summary:
In some recent CEs, we had difficulty figuring out why a node was working slowly. In part this was because we only log operations as being slow that we expect might be slow; those operations tend to do a lot of complicated things like I/O, kernel calls, and memory allocation. Unfortunately this means we can't tell from the fact that they are running slow what underlying resource is actually slowing us down.
This diff adds logging that checks only one thing: can the reactor run scheduled tasks roughly at the time scheduled? This essentially tests how long it takes to get a thread running on a core once it becomes Runnable.
More precisely, we instrument PeriodicTimer so when its callback gets run we compare the time it was supposed to run at to now and warn if more than a threshold of time has elapsed after it was supposed to run. (We don't particularly care if it runs early although that is unlikely to happen in practice.)
A new gflag controls the threshold; by default it is 20 ms which is quite generous.
Note that the main PeriodicTimer's in use are those for determining when to call a Raft election (3 seconds using default gflags) and when a Raft heartbeat needs to be sent (500 ms).
Note that the Linux scheduler is pretty responsive for "interactive threads" like the reactor threads; simply pegging CPU will not necessarily hurt scheduling occasional work on such threads that much.
Fixes #24801
Jira: DB-13902
Test Plan:
Simultaneously while running the following test:
```
ybd release --cxx-test xcluster-test --gtest_filter '*.NoCleanupOfTransactionsInFlushedFiles'
```
run the following stress command which starts high priority threads to use all the cores:
```
sudo stress-ng --cpu 0 --sched fifo --sched-prio 99 --timeout 10s
```
Sample log lines from this diff for this:
```
[C-ts-2] W1021 16:29:10.464721 4537 periodic.cc:175] PeriodicTimer callback delayed by 0.745s
[C-ts-1] W1021 16:29:12.444103 4483 periodic.cc:175] PeriodicTimer callback delayed by 0.881s [suppressed 40 similar messages]
[C-ts-2] W1021 16:29:13.444571 4537 periodic.cc:175] PeriodicTimer callback delayed by 0.555s [suppressed 24 similar messages]
[P-ts-1] W1021 16:29:14.444896 4261 periodic.cc:175] PeriodicTimer callback delayed by 0.533s [suppressed 15 similar messages]
[P-ts-1] W1021 16:29:15.444927 4263 periodic.cc:175] PeriodicTimer callback delayed by 0.576s [suppressed 19 similar messages]
[P-ts-2] W1021 16:29:16.444965 4317 periodic.cc:175] PeriodicTimer callback delayed by 0.730s [suppressed 15 similar messages]
[P-ts-2] W1021 16:29:17.445015 4317 periodic.cc:175] PeriodicTimer callback delayed by 0.749s [suppressed 22 similar messages]
[P-ts-3] W1021 16:29:19.444669 4373 periodic.cc:175] PeriodicTimer callback delayed by 0.590s [suppressed 13 similar messages]
```
Reviewers: asrivastava
Reviewed By: asrivastava
Subscribers: ybase
Differential Revision: https://phorge.dev.yugabyte.com/D47652