Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Properly handle unknown chunk sizes in P2P rechunking #7856

Merged
merged 13 commits into from
May 25, 2023
Merged

Properly handle unknown chunk sizes in P2P rechunking #7856

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
b3cf8d0
Rename
hendrikmakait May 24, 2023
f5226b3
[skip-caching]
hendrikmakait May 24, 2023
6814398
Merge branch 'main' of https://github.com/dask/distributed into renam…
jrbourbeau May 24, 2023
a10e1eb
[skip-caching]
jrbourbeau May 24, 2023
62a896a
Try invalidating cache [skip-caching]
jrbourbeau May 24, 2023
b6503ef
Align tests with dask/dask#10027
hendrikmakait May 25, 2023
85170e0
Align tests with dask/dask#10003
hendrikmakait May 25, 2023
9728331
Align tests with dask/dask#10157
hendrikmakait May 25, 2023
1044b51
Fix handling of unknown chunk sizes
hendrikmakait May 25, 2023
16ea91d
Imports
hendrikmakait May 25, 2023
a7ae007
Skip tests due to performance problems
hendrikmakait May 25, 2023
f681720
Comment
hendrikmakait May 25, 2023
b6fe358
pre-commit
hendrikmakait May 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,5 +63,5 @@ repos:
- pytest
- tornado
- pyarrow
- git+https://github.com/dask/dask
- git+https://github.com/dask/dask.git
hendrikmakait marked this conversation as resolved.
Show resolved Hide resolved
- git+https://github.com/dask/zict
4 changes: 2 additions & 2 deletions distributed/shuffle/_worker_extension.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -301,7 +301,7 @@
memory_limiter_disk: ResourceLimiter,
memory_limiter_comms: ResourceLimiter,
):
from dask.array.rechunk import _old_to_new
from dask.array.rechunk import old_to_new

Check warning on line 304 in distributed/shuffle/_worker_extension.py

View check run for this annotation

Codecov / codecov/patch

distributed/shuffle/_worker_extension.py#L304 

Added line #L304 was not covered by tests

super().__init__(
id=id,
Expand All @@ -323,7 +323,7 @@
self.partitions_of = dict(partitions_of)
self.worker_for = worker_for
self._slicing = rechunk_slicing(old, new)
self._old_to_new = _old_to_new(old, new)
self._old_to_new = old_to_new(old, new)

Check warning on line 326 in distributed/shuffle/_worker_extension.py

View check run for this annotation

Codecov / codecov/patch

distributed/shuffle/_worker_extension.py#L326 

Added line #L326 was not covered by tests

async def _receive(self, data: list[tuple[ArrayRechunkShardID, bytes]]) -> None:
self.raise_if_closed()
Expand Down