Introduce PreloadManager to handle failures in preload setup/teardown

[hendrikmakait]

This PR introduces a PreloadManager which deduplicates preload handling code from the Client, Nanny, Scheduler, and Worker. The PreloadManager also ensures that errors on preload.{start|teardown} don't raise.

• Tests added / passed
• Passes pre-commit run --all-files

[hendrikmakait]

There's an argument to be made that preload.start should raise here. Thoughts?

[crusaderky]

Agreed; I don't think it's healthy to semi-quietly go online nonetheless.

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

       21 files  +       1         21 suites  +1   10h 49m 43s ⏱️ - 29m 17s
  3 772 tests +     18    3 657 ✔️ +     12     109 💤 +  3  6 ❌ +4 
35 207 runs   - 1 107  33 439 ✔️  - 1 121  1 761 💤 +12  7 ❌ +3 

For more details on these failures, see this check.

Results for commit 420ac66. ± Comparison against base commit 9255987.

This pull request removes 14 and adds 32 tests. Note that renamed tests count towards both.

distributed.tests.test_cancelled_state ‑ test_execute_preamble_early_cancel[executing-False-deserialize_task]
distributed.tests.test_cancelled_state ‑ test_execute_preamble_early_cancel[executing-False-execute]
distributed.tests.test_cancelled_state ‑ test_execute_preamble_early_cancel[executing-True-deserialize_task]
distributed.tests.test_cancelled_state ‑ test_execute_preamble_early_cancel[executing-True-execute]
distributed.tests.test_cancelled_state ‑ test_execute_preamble_early_cancel[resumed-False-deserialize_task]
distributed.tests.test_cancelled_state ‑ test_execute_preamble_early_cancel[resumed-False-execute]
distributed.tests.test_cancelled_state ‑ test_execute_preamble_early_cancel[resumed-True-deserialize_task]
distributed.tests.test_cancelled_state ‑ test_execute_preamble_early_cancel[resumed-True-execute]
distributed.tests.test_preload ‑ test_failure_doesnt_crash
distributed.tests.test_scheduler ‑ test_dumps_task
…

distributed.cli.tests.test_dask_worker.test_listen_address_ipv6[tcp:..[ ‑ 1]:---nanny]
distributed.cli.tests.test_dask_worker.test_listen_address_ipv6[tcp:..[ ‑ 1]:---no-nanny]
distributed.diagnostics.tests.test_memray
distributed.diagnostics.tests.test_memray ‑ test_basic_integration_scheduler
distributed.diagnostics.tests.test_memray ‑ test_basic_integration_scheduler_report_args[False]
distributed.diagnostics.tests.test_memray ‑ test_basic_integration_scheduler_report_args[report_args0]
distributed.diagnostics.tests.test_memray ‑ test_basic_integration_workers[1]
distributed.diagnostics.tests.test_memray ‑ test_basic_integration_workers[False]
distributed.diagnostics.tests.test_memray ‑ test_basic_integration_workers[True]
distributed.diagnostics.tests.test_memray ‑ test_basic_integration_workers_report_args[False]
…

♻️ This comment has been updated with latest results.

[crusaderky]

LGTM. I suggest to leave the change to crash in case of failing preload start to a separate PR.