Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: support placeholders in shape-only routines #2652

Merged
merged 6 commits into from
Aug 23, 2023

Conversation

agoose77
Copy link
Collaborator

@agoose77 agoose77 commented Aug 16, 2023

When dask-awkward used its column optimisation to generate placeholder arrays, we were seeing placeholders in some nplike functions that were not prepared for them. This is because these nplike functions explicitly don't touch data for typetracers.

This fix won't solve the re-hydration issues by itself, but it should get us closer and is fairly easy to reason about.

@codecov
Copy link

codecov bot commented Aug 16, 2023

Codecov Report

Merging #2652 (16f20a1) into main (0accfad) will decrease coverage by 0.07%.
The diff coverage is 43.47%.

Additional details and impacted files
Files Changed Coverage Δ
src/awkward/_nplikes/jax.py 79.54% <0.00%> (ø)
src/awkward/_nplikes/array_module.py 80.86% <30.00%> (-8.02%) ⬇️
src/awkward/_nplikes/placeholder.py 48.88% <71.42%> (ø)
src/awkward/_nplikes/numpylike.py 73.76% <75.00%> (+0.01%) ⬆️
src/awkward/_nplikes/typetracer.py 76.17% <75.00%> (-0.13%) ⬇️

@agoose77 agoose77 temporarily deployed to docs-preview August 16, 2023 20:43 — with GitHub Actions Inactive
@agoose77 agoose77 marked this pull request as ready for review August 17, 2023 12:58
@agoose77 agoose77 temporarily deployed to docs-preview August 17, 2023 13:12 — with GitHub Actions Inactive
@agoose77 agoose77 temporarily deployed to docs-preview August 22, 2023 11:24 — with GitHub Actions Inactive
@agoose77
Copy link
Collaborator Author

Amongst other things, this PR adds nbytes as a part of the ArrayLike protocol. TypeTracerArray will use the shape to return a maybe-unknown number of bytes (length), whilst PlaceholderArray returns 0. This is because placeholder arrays are explicitly data-less, whilst typetracers are stand-in arrays.

Copy link
Member

@jpivarski jpivarski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

for item in reversed(x._shape):
for item in x._shape[-1:0:-1]:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not just style: it's a change in behavior. Was the original wrong?

We talked about this on Zoom, and yes—it used to be wrong. Now it's right.

@jpivarski jpivarski enabled auto-merge (squash) August 23, 2023 14:34
@jpivarski jpivarski temporarily deployed to docs-preview August 23, 2023 14:48 — with GitHub Actions Inactive
@jpivarski jpivarski merged commit 8feae62 into main Aug 23, 2023
34 checks passed
@jpivarski jpivarski deleted the agoose77/fix-placeholder-support branch August 23, 2023 14:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants