Fix restart wait for workers edge case

[gjoseph92]

restart would poll for new workers on a 0.2s interval. If a worker joined in the 0.2s after the final check, but before the timeout had expired, we'd raise a confusing error claiming not all workers had returned, but also demonstrating that they had.

Closes #6821

• Tests added / passed
• Passes pre-commit run --all-files

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

       15 files  ±0         15 suites  ±0   6h 15m 56s ⏱️ - 6m 0s
  2 989 tests +1    2 897 ✔️  - 1       89 💤 +1  2 ❌ ±0  1 🔥 +1 
22 165 runs  +8  21 115 ✔️ +5  1 046 💤 +1  3 ❌ +1  1 🔥 +1 

For more details on these failures and errors, see this check.

Results for commit f8188d6. ± Comparison against base commit 4f6960a.

[hendrikmakait]

LGTM, good catch!