[tune] Reduce default number of maximum pending trials to max(16, cluster_cpus)

[krfricke]

Why are these changes needed?

The default number of 1000 pending trials for random search leads to unnecessary logs being produced and scheduling overhead, as well as clogging and growing the gcs_server.out logfile to gigabytes when running long running trials.

1000 trials were chosen to ensure that autoscaling can be aggressively triggered. This PR instead sets a minimum of 16 pending trials to enforce a lower bound of autoscaling behavior. If the number of available cluster CPUs is higher than this number, the number of available CPUs is used instead. If this is a high number, a warning is raised.

To trigger more aggressive autoscaling behavior, users will have to set the TUNE_MAX_PENDING_TRIALS_PG environment variable.

cc @ericl @max0x7ba

Related issue number

Addresses #15504
Addresses #15595
Addresses https://discuss.ray.io/t/ray-tune-log-grows-extremely-large/1992/3

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(