Skip to content

Fix argtopk with uneven chunks#3720

Merged
mrocklin merged 8 commits intodask:masterfrom
crusaderky:argtopk_chunks
Jul 8, 2018
Merged

Fix argtopk with uneven chunks#3720
mrocklin merged 8 commits intodask:masterfrom
crusaderky:argtopk_chunks

Conversation

@crusaderky
Copy link
Copy Markdown
Collaborator

Closes #3719.

The PR is complete for the specific trivial problem. However I'd also like to change normalize_chunks to prevent similar problems in the future - see discussion in #3719.

@mrocklin
Copy link
Copy Markdown
Member

mrocklin commented Jul 5, 2018

This looks good to me. The test failure is I suspect due to an upstream dependency shift. It looks like pillow updated a few days ago and probably no longer supports not having docstrings in optimized mode. This should probably be handled as a separate issue though.

I've just pushed #3723

@crusaderky
Copy link
Copy Markdown
Collaborator Author

Same problem here as in the other build - https://travis-ci.org/dask/dask/jobs/400540686#L1337-L1352

@mrocklin
Copy link
Copy Markdown
Member

mrocklin commented Jul 5, 2018

Yeah, my guess is that the recent update of pillow caused something odd to happen. We should probably report this upstream.

@mrocklin
Copy link
Copy Markdown
Member

mrocklin commented Jul 6, 2018

The CI failure has been patched up for now. I believe that you may have to merge with master though.

chunks = ((0,),) * len(shape)

if (shape and len(shape) == 1 and len(chunks) > 1 and
all(type(c) is not tuple for c in chunks)):
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be ok to replace type(c) is not tuple with isinstance(c, numbers.Number)? That way we'll probably continue doing something sensible if they pass a list like chunks=[[1], [2], [3], [4]]. Obviously this isn't recommended, but new users can do all sorts of things :)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[[1], [2], [3], [4]] is already covered by the unit tests. Anyway I changed it.

with pytest.raises(ValueError):
normalize_chunks(((10,), ), (11, ))
with pytest.raises(ValueError):
normalize_chunks(((5, ), (5, )), (5, ))
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the extra tests here

from dask.array.core import (
normalize_chunks as _normalize_chunks,
)
from dask.array.core import normalize_chunks
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that makes sense to me. Thanks

@mrocklin
Copy link
Copy Markdown
Member

mrocklin commented Jul 8, 2018

This seems good to me. I had one minor comment, but would be happy ignoring it as well.

@crusaderky
Copy link
Copy Markdown
Collaborator Author

@mrocklin ready for merge

@mrocklin mrocklin merged commit e279e8c into dask:master Jul 8, 2018
@mrocklin
Copy link
Copy Markdown
Member

mrocklin commented Jul 8, 2018

Thanks @crusaderky ! Merging.

@crusaderky crusaderky deleted the argtopk_chunks branch July 24, 2018 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants