[Adaptive] Do not allow workers to downscale if they are running long-running tasks (e.g. worker_client)

[fjetter]

Having tasks flagged as long running protects the worker from being classified as idle. However, when scaling adaptively and Adapt recommends fewer workers than there are idle, this can cause workers with seceded tasks to be downscaled even though there are better options.

This suggests to only close workers with long running tasks if there are no other possible options.

Considering that these workers may be driving a computation we may even need to go as far as to prohibit downscaling of those tasks entirely

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

       27 files  ±  0         27 suites  ±0   11h 46m 14s ⏱️ + 13m 13s
  3 942 tests +  2    3 832 ✔️ +  2     108 💤 ±0  2 ❌ ±0 
49 584 runs  +26  47 312 ✔️ +25  2 270 💤 +1  2 ❌ ±0 

For more details on these failures, see this check.

Results for commit 315b337. ± Comparison against base commit 415d4fa.

♻️ This comment has been updated with latest results.

[mrocklin]

Considering that these workers may be driving a computation we may even need to go as far as to prohibit downscaling of those tasks entirely

I think that if there is a running task, long-running or otherwise, that we should not choose to remove that worker when using adaptive scaling, even if it is mostly-idle.

[fjetter]

I think that if there is a running task, long-running or otherwise, that we should not choose to remove that worker when using adaptive scaling, even if it is mostly-idle.

I think I agree given the current implementation. There is a possibility to allow for workers to naturally conclude their work by not allowing further assignments of tasks. This would be a similar mechanism as suggested in #3761 (comment) (basically setting a worker to pause and steal the remaining tasks)
For now, I think I don't want to go down that path but merely want to show options if a stricter behavior causes problems.

[fjetter]

this now prohibits downscaling of workers with long-running tasks, e.g. tasks-within-tasks

[crusaderky]

Mostly nits