Ensure we test against numpy 2 in CI

[jrbourbeau]

NumPy 2 came out over the weekend. We've been testing against the nightly version of numpy in our upstream build but due to other dependencies, numpy >=2 isn't getting installed on our main CI environments.

This PR removes some optional dependencies (e.g. numba) to ensure numpy 2 can be installed in our Python 3.12 builds.

[jrbourbeau]

Okay, so there are some doctest failures (I'll push up a fix for that) and a single test failure

FAILED dask/tests/test_tokenize.py::test_tokenize_random_functions[np.random] - AssertionError: assert '5a758251ee05...fadb5e734ae74' == 'f8c814ecbf02...12210d58c137e'
  
  - f8c814ecbf0209a0c4812210d58c137e
  + 5a758251ee0551802acfadb5e734ae74

@fjetter do you have a moment to look at that test failure? It was added in #10865, which you reviewed

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

     15 files  ±  0       15 suites  ±0   3h 22m 19s ⏱️ - 1m 17s
 13 126 tests +  2   12 148 ✅ +  2     931 💤 ± 0  47 ❌ ±0 
162 260 runs   - 272  142 083 ✅  - 370  20 130 💤 +98  47 ❌ ±0 

For more details on these failures, see this check.

Results for commit 48a9b8a. ± Comparison against base commit 36e9d7c.

♻️ This comment has been updated with latest results.

[jrbourbeau]

Manually setting the random seed here gets this test passing. If we don't do this, then this test fails against NumPy 2 with this tokenization mismatch error:

__________________ test_tokenize_random_functions[np.random] ___________________
[gw0] linux -- Python 3.12.4 /home/runner/miniconda3/envs/test-environment/bin/python3.12

module = <module 'numpy.random' from '/home/runner/miniconda3/envs/test-environment/lib/python3.12/site-packages/numpy/random/__init__.py'>

    @pytest.mark.parametrize(
        "module",
        ["random", pytest.param("np.random", marks=pytest.mark.skipif("not np"))],
    )
    def test_tokenize_random_functions(module):
        """random.random() and other methods of the global random state do not compare as
        equal to themselves after a pickle roundtrip"""
        module = eval(module)
    
        a = module.random
        b = pickle.loads(pickle.dumps(a))
>       assert check_tokenize(a) == check_tokenize(b)
E       AssertionError: assert '5a758251ee05...fadb5e734ae74' == 'f8c814ecbf02...12210d58c137e'
E         
E         - f8c814ecbf0209a0c4812210d58c137e
E         + 5a758251ee0551802acfadb5e734ae74

dask/tests/test_tokenize.py:1277: AssertionError

@fjetter @phofl @hendrikmakait do any of you have more context here?