Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose BitSliceIterator and BitIndexIterator (#1864) #1865

Merged
merged 4 commits into from
Jun 16, 2022

Conversation

tustvold
Copy link
Contributor

@tustvold tustvold commented Jun 13, 2022

Which issue does this PR close?

Part of #1864

Rationale for this change

We have heavily optimised code for iterating bitmasks, we should use it in more places

What changes are included in this PR?

Extracts the core logic of filter::SlicesIterator and filter::IndexIterator into a public module where they can be used by other code.

I have confirmed there is no impact on the performance of the filter benchmarks

Are there any user-facing changes?

No

@github-actions github-actions bot added the arrow Changes to the arrow crate label Jun 13, 2022
Comment on lines +118 to +119
let next = self.iter.next().expect("IndexIterator exhausted early");
self.remaining -= 1;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fascinatingly swapping these two lines around results in a 50% performance regression for some benchmarks

@codecov-commenter
Copy link

codecov-commenter commented Jun 13, 2022

Codecov Report

Merging #1865 (052461b) into master (3073a26) will decrease coverage by 0.02%.
The diff coverage is 81.81%.

❗ Current head 052461b differs from pull request most recent head d2e5760. Consider uploading reports for the commit d2e5760 to get more accurate results

@@            Coverage Diff             @@
##           master    #1865      +/-   ##
==========================================
- Coverage   83.48%   83.45%   -0.03%     
==========================================
  Files         201      203       +2     
  Lines       57000    57145     +145     
==========================================
+ Hits        47584    47688     +104     
- Misses       9416     9457      +41     
Impacted Files Coverage Δ
arrow/src/csv/writer.rs 71.69% <ø> (ø)
arrow/src/datatypes/native.rs 66.66% <ø> (ø)
arrow/src/ffi.rs 86.97% <ø> (ø)
arrow/src/pyarrow.rs 0.00% <0.00%> (ø)
arrow/src/util/bit_chunk_iterator.rs 93.83% <ø> (ø)
parquet/src/arrow/record_reader/buffer.rs 92.42% <ø> (ø)
...et/src/arrow/array_reader/byte_array_dictionary.rs 83.91% <33.33%> (ø)
arrow/src/buffer/mutable.rs 89.13% <37.50%> (-1.41%) ⬇️
arrow/src/array/data.rs 84.16% <50.00%> (ø)
arrow/src/array/builder.rs 86.48% <65.51%> (-0.41%) ⬇️
... and 21 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3073a26...d2e5760. Read the comment docs.

tustvold and others added 2 commits June 16, 2022 20:10
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
@alamb alamb changed the title Expose BitSliceIterator and BitIndexIterator (#1864) Expose BitSliceIterator and BitIndexIterator (#1864) Jun 16, 2022
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tustvold tustvold merged commit c1525a4 into apache:master Jun 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants