Summary:
Problem
All the threads on a tserver may be exhausted trying to do a remote bootstrap. without any progress.
This is the sequence of events that led to it:

  # A remote bootstrap was triggered for a tablet from a peer (normal)
  # An RPC was sent to the affected tserver as a first step to initiate a checkpoint
  # A thread was created to process this RPC. It acquired the checkpoint lock (mutex) but got stuck in the IO (abnormal, rootcause of the incident).
  # The RPC timed out and remote bootstrap failed
  # New RPC was sent again to the same tserver to repeat the failed remote bootstrap
  # Another thread was created to process the second RPC. It got stuck waiting for the checkpoint lock.
  # The new RPC timed out and remote bootstrap failed again
  # Steps 5 and 6 were repeated until all threads were exhausted on the tserver.

Solution
Use a try_lock API on the checkpoint lock (mutex). If the thread fails to acquire it, RPC fails and returns an error.
This way there are at most two threads active for remote bootstrap RPC - one stuck doing it, other trying to do it.

It does not save us the wasted repetitive overhead of the network calls (RPC) or launching the 2nd thread repeatedly.
Assuming this slow bootstrap initiation is rare enough that it should be okay to pay this cost.

More details: https://docs.google.com/document/d/1siqRCNGlXkOCDIzibZoEa6ArpNKgLBgAYKMOAoDFwmE/edit?usp=sharing

Original commit: bd1f156a6d5ccadf48b098d604f4906dc5c3ae69 / D50240

Test Plan:
./yb_build.sh release --cxx-test tserver_remote_bootstrap_service-test --gtest_filter *Checkpoint*
./yb_build.sh release --cxx-test integration-tests_remote_bootstrap-itest --gtest_filter *Checkpoint*

Reviewers: hsunder, arybochkin, neera.mital, #db-approvers

Reviewed By: neera.mital, #db-approvers

Subscribers: svc_phabricator, ybase, arybochkin

Differential Revision: https://phorge.dev.yugabyte.com/D55827