Ensure restart clears taskgroups et al

[fjetter]

Restart keeps TaskGroups, TaskPrefixes, etc. around.

This can have a non-trivial impact on scheduling, e.g.

• distributed/distributed/scheduler.py

  Line 2648 in c15a10e

  duration: float = ts.prefix.duration_average

• distributed/distributed/scheduler.py

  Lines 1802 to 1805 in c15a10e

  	valid_workers is None
  	and len(tg) > self.total_nthreads * 2
  	and len(tg.dependencies) < 5
  	and sum(map(len, tg.dependencies)) < 5

• distributed/distributed/scheduler.py

  Lines 2015 to 2022 in c15a10e

  	s: set = self.unknown_durations.pop(ts.prefix.name, set())
  	tts: TaskState
  	steal = self.extensions.get("stealing")
  	for tts in s:
  	if tts.processing_on:
  	self._set_duration_estimate(tts, tts.processing_on)
  	if steal:
  	steal.recalculate_cost(tts)

and there are possibly more collections where this matters. In a perfect world, all mutable state would be encapsulated in a dedicated class of which we could create a new instance in case of a restart but we're very far away from this. I think these are the most important collections to clear.

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

       15 files  ±0         15 suites  ±0   6h 40m 20s ⏱️ + 13m 19s
  3 041 tests ±0    2 955 ✔️ +3    83 💤  - 1  3 ❌  - 2 
22 493 runs  ±0  21 518 ✔️ +4  972 💤  - 2  3 ❌  - 2 

For more details on these failures, see this check.

Results for commit 35db050. ± Comparison against base commit c15a10e.

♻️ This comment has been updated with latest results.

[crusaderky]

You're creating a new reference to the same WeakSet. The assertion below is always true because it's the same object.
Also, I suspect this may be prone to race conditions with late/delayed cleanup from previous tests?

[fjetter]

you're right of course. I don't even know if this is a sane test, to be honest. We've been looking into TaskState garbage collection before and this is a bit tricky. I'll probably just remove this

[crusaderky]

.

[ian-r-rose]

Very interested to see if this has an effect on coiled/benchmarks#253

[ian-r-rose]

This does indeed seem to fix coiled/benchmarks#253. The test_climatic_mean test passes, with considerably better memory usage:

Average memory for that test module:

Duration:

Interestingly, it doesn't seem to have an enormous effect on other tests.