fix: use ak.merge_union_of_records to generate input data format #1017
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lgray
force-pushed
the
use_merge_union_of_records
branch
from
5f638d2
to
6e3e492
Compare
lgray
force-pushed
the
use_merge_union_of_records
branch
from
aca12a2
to
d2b10ef
Compare
|
@nsmith- this work around was the best I could come up with for now. This at least lets us deal with changing forms in a dataset in a way that doesn't mess up the I think it keeps the leak fairly contained, but if you can think of a cleaner solution I'll happily accept it. |
lgray
force-pushed
the
use_merge_union_of_records
branch
from
d2b10ef
to
a1dbafe
Compare
Why not |
|
Because I can't get it to work properly. |
|
After letting this simmer all day in the back of my head I think I have a way to get it to do option arrays. I'll give it a try. |
|
@nsmith- ready for a review. |
lgray
force-pushed
the
use_merge_union_of_records
branch
3 times, most recently
from
1b0b518
to
0d9fe7c
Compare
lgray
force-pushed
the
use_merge_union_of_records
branch
from
0eef99b
to
bf40012
Compare
nsmith-
approved these changes
Feb 2, 2024
lgray
force-pushed
the
use_merge_union_of_records
branch
4 times, most recently
from
974c56b
to
51fbc80
Compare
…present computed form
lgray
force-pushed
the
use_merge_union_of_records
branch
from
51fbc80
to
7f82629
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1014
May not need to merge if current pre-processing scheme already functions for this task.As it is right now this PR uses ak.merge_union_of_records to tag fields that are not shared across files in a dataset.
When a file is parsed, if a given key is not available in that file a buffer is generated as an array of
Noneusing an IndexedOptionArray and the expected length of the step in that form key.The only kind of arrays that are supported by this workaround at present are flat arrays of booleans. The user may choose with
ak.fill_nonethe default behavior for the boolean to match their needs.If any other kind of array is encountered the file is considered not openable as nanoevents and emits errors either in preprocessing or within nanoevents itself.