Handle null partitions in P2P shuffling #8116
Conversation
| @@ -2057,6 +2058,28 @@ async def test_handle_null_partitions_p2p_shuffling(c, s, *workers): | |||
| await check_scheduler_cleanup(s) | |||
|
|
|||
|
|
|||
| @gen_cluster(client=True) | |||
| async def test_handle_null_partitions_p2p_shuffling_2(c, s, a, b): | |||
There was a problem hiding this comment.
This test fails only fails 2/3 of the time on main, but that's good enough for me.
| def make_partition(i): | ||
| """Return null column for one partition""" | ||
| if i % 2 == 1: | ||
| return pd.DataFrame({"a": np.random.random(10), "b": [None] * 10}) |
There was a problem hiding this comment.
I would make the None column null explicitly, otherwise lgtm
There was a problem hiding this comment.
This fails unless I replace b in the other case with a string. That's another issue I plan to handle. As the current test mimics the original reproducer, I will keep this.
Unit Test ResultsSee test report for an extended history of previous test failures. This is useful for diagnosing flaky tests. 21 files ±0 21 suites ±0 11h 36m 28s ⏱️ - 5m 4s For more details on these failures, see this check. Results for commit 6243868. ± Comparison against base commit aab76d7. |
Closes #8092
Follow-up to #7922
pre-commit run --all-files