Run saltcheck tests in parallel

[mchugh19]

What does this PR do?

Adds multiprocessing capability to saltcheck. I'm not sure if this is worth the additional complexity or if this is the way the core-team would like multiprocessing supported. Thoughts?

Tests will be run in parallel by adding saltcheck_parallel: True in minion config.
Setting this value to an integer will set the maximum parallel processes.

For each state, tests are multiprocessed batched by
Optional value set in saltcheck_parallel
or
The minimum between the number of processors or the number of tests for the state

Previous Behavior

Given tests

test time:
  module_and_function: test.sleep
  args:
    - 3
  assertion: assertTrue

test time 2:
  module_and_function: test.sleep
  args:
    - 3
  assertion: assertTrue

test time 3:
  module_and_function: test.sleep
  args:
    - 3
  assertion: assertTrue

test time 4:
  module_and_function: test.sleep
  args:
    - 3
  assertion: assertTrue

test time 5:
  module_and_function: test.sleep
  args:
    - 3
  assertion: assertTrue

test time 6:
  module_and_function: test.sleep
  args:
    - 3
  assertion: assertTrue

and

from second:
  module_and_function: test.sleep
  args:
    - 3
  assertion: assertTrue

from second2:
  module_and_function: test.sleep
  args:
    - 3
  assertion: assertTrue

saltcheck processing runs sequentially

      TEST RESULTS:
          ----------
          Execution Time:
              28.9358
          Failed:
              0
          Missing Tests:
              0
          Passed:
              8
          Skipped:
              0

real    0m31.508s
user    0m6.565s
sys     0m0.794s

26 seconds of sleep completes in 31 seconds total

New Behavior

Same tests now finish faster

      TEST RESULTS:
          ----------
          Execution Time:
              59.6738
          Failed:
              0
          Missing Tests:
              0
          Passed:
              8
          Skipped:
              0

real    0m17.431s
user    0m36.822s
sys     0m1.494s

Same 26 seconds of sleep completes in 17 seconds (individual test duration time is no longer very accurate as it includes multiprocessing overhead and since many tests run in parallel, the sum into execution time is even more off (especially for a series of short tests))
Adjusting the sleeps to larger values totaling 59 seconds, completes in 30 seconds

Tests written?

No
Not sure how to write a test as the output is unchanged. Is it possible to write assertions against the debug log?

Commits signed with GPG?

No

[mchugh19]

re-run pr-windows2016-py3

[mchugh19]

re-run pr-pre-commit

[dwoz]

@mchugh19 I'm not a fan of overloading the meaning of config values or allowing them to have multiple types. I'd prefer that saltcheck_parallel remain a boolean and use a second config value for the number of potential processes, that value can default to number of available processes.

[mchugh19]

I'm not a fan of overloading the meaning of config values or allowing them to have multiple types. I'd prefer that saltcheck_parallel remain a boolean and use a second config value for the number of potential processes, that value can default to number of available processes.

Sure. That's easy enough. Maybe you could look over #56101 first, since either will need to be rebased after the other one is merged anyway.

[mchugh19]

@dwoz as suggested, another saltcheck_processes parameter is now used to limit parallelization. Once tests pass, this should be good to go.