Avoid sorting the queued heap

[fjetter]

I saw this crippling CPU on the scheduler for cases where there are many tasks queued (>250k). I haven't thoroughly tested this but I don't see the need to sort. I also believe there is no need to transition all at once since individual task chains would be processed before the next queued tasks anyhow, i.e. this should not affect ordering, I believe.

Generally, the sort does not truly help us here and we have to pop the heap for every task individually anyhow.

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

       20 files   -        1         20 suites   - 1   10h 31m 47s ⏱️ + 4m 26s
  3 849 tests +     13    3 739 ✔️ +     15     107 💤 ±  0    3 ❌  - 2 
35 828 runs   - 1 243  34 084 ✔️  - 1 177  1 734 💤  - 70  10 ❌ +4 

For more details on these failures, see this check.

Results for commit 1a661a4. ± Comparison against base commit de3f755.

This pull request removes 9 and adds 22 tests. Note that renamed tests count towards both.

distributed.tests.test_worker ‑ test_conda_install
distributed.tests.test_worker ‑ test_conda_install_fails_on_returncode
distributed.tests.test_worker ‑ test_conda_install_fails_when_conda_not_found
distributed.tests.test_worker ‑ test_conda_install_fails_when_conda_raises
distributed.tests.test_worker ‑ test_package_install_failing_does_not_restart_on_nanny
distributed.tests.test_worker ‑ test_package_install_installs_once_with_multiple_workers
distributed.tests.test_worker ‑ test_package_install_restarts_on_nanny
distributed.tests.test_worker ‑ test_pip_install
distributed.tests.test_worker ‑ test_pip_install_fails

distributed.diagnostics.tests.test_packageinstall ‑ test_conda_install
distributed.diagnostics.tests.test_packageinstall ‑ test_conda_install_fails_on_returncode
distributed.diagnostics.tests.test_packageinstall ‑ test_conda_install_fails_when_conda_not_found
distributed.diagnostics.tests.test_packageinstall ‑ test_conda_install_fails_when_conda_raises
distributed.diagnostics.tests.test_packageinstall ‑ test_package_install_failing_does_not_restart_on_nanny
distributed.diagnostics.tests.test_packageinstall ‑ test_package_install_installs_once_when_reregistered
distributed.diagnostics.tests.test_packageinstall ‑ test_package_install_installs_once_with_multiple_workers
distributed.diagnostics.tests.test_packageinstall ‑ test_package_install_restarts_on_nanny
distributed.diagnostics.tests.test_packageinstall ‑ test_pip_install
distributed.diagnostics.tests.test_packageinstall ‑ test_pip_install_fails
…

♻️ This comment has been updated with latest results.

[crusaderky]

Good catch!
Only a small comment.