Support for Python 3.12

[graingert]

Closes #xxxx

• Tests added / passed
• Passes pre-commit run --all-files

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

       25 files  +       4         25 suites  +4   13h 27m 18s ⏱️ + 3h 8m 27s
  3 858 tests ±       0    3 737 ✔️ +       2     117 💤 ±    0    4 ❌  - 1 
44 927 runs  +7 627  42 779 ✔️ +7 313  2 122 💤 +310  26 ❌ +5 

For more details on these failures, see this check.

Results for commit cfbc37a. ± Comparison against base commit b49c0bb.

♻️ This comment has been updated with latest results.

[crusaderky]

I've moved several unrelated changes to different PRs.
Outstanding comments below:

[crusaderky]

I've reverted the gilknocker changes and moved them to #8263.

I've also reverted the pyarrow import changes and was going to open a separate PR for them (main...crusaderky:distributed:no_pyarrow) but then I realised that pyarrow is a dependency in the dask conda package, which in turn is a dependency of so many other conda packages, so I'm not sure if we should bother dealing with the use case? A user would need to run the tests with pandas but with a strictly pip-installed version of dask and I wonder if anyone cares. CC @hendrikmakait for opinion

[hendrikmakait]

I've also reverted the pyarrow import changes and was going to open a separate PR for them (main...crusaderky:distributed:no_pyarrow) but then I realised that pyarrow is a dependency in the dask conda package, which in turn is a dependency of so many other conda packages, so I'm not sure if we should bother dealing with the use case? A user would need to run the tests on a pure pypi environment and I doubt anyone cares.

I doubt anybody cares right now, but it would be nice if setting up a Conda environment would not be a prerequisite for contributing to Dask. If this were a major maintenance burden I'd say let's scratch that, but since we have the patch ready, I see no reason for dropping the support for this right now.

[crusaderky]

I've also reverted the pyarrow import changes and was going to open a separate PR for them (main...crusaderky:distributed:no_pyarrow) but then I realised that pyarrow is a dependency in the dask conda package, which in turn is a dependency of so many other conda packages, so I'm not sure if we should bother dealing with the use case? A user would need to run the tests on a pure pypi environment and I doubt anyone cares.

I doubt anybody cares right now, but it would be nice if setting up a Conda environment would not be a prerequisite for contributing to Dask. If this were a major maintenance burden I'd say let's scratch that, but since we have the patch ready, I see no reason for dropping the support for this right now.

Actually, it's not that simple. Let's move the discussion to #8265.

[crusaderky]

Current status

• Blocked by Use memoryview.nbytes when warning on large graph send #8268
• One outstanding comment on nanny.py