Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove selective unfiltering. #2410

Merged
merged 1 commit into from
Jul 25, 2021

Conversation

KiterLuc
Copy link
Contributor

@KiterLuc KiterLuc commented Jul 12, 2021

From what we know about our customer's use case, they either will do the full array where this gains nothing (and actually this affects the performance negatively because of the added code complexity), or very targeted ones, for which the gains would be very small, unless the tile capacity/extent is unusually large if the users have a poorly configured array.

Also, this will allow us to implement compression codec for video and imaging, which won't work with selective unfiltering.


TYPE: IMPROVEMENT
DESC: Remove selective unfiltering.

@shortcut-integration
Copy link

This pull request has been linked to Clubhouse Story #8874: Remove selective unfiltering.

@KiterLuc KiterLuc force-pushed the lr/remove-selective-unfiltering/ch8874 branch 2 times, most recently from 1d19672 to 720674d Compare July 13, 2021 15:06
@KiterLuc KiterLuc force-pushed the lr/remove-selective-unfiltering/ch8874 branch from 720674d to d5ffd4d Compare July 21, 2021 17:02
Copy link
Member

@stavrospapadopoulos stavrospapadopoulos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, but will we have backward compatibility issues? cc @Shelnutt2

RETURN_NOT_OK(
output->init_var_size(buffer_addressing, std::move(chunk_sizes)));
RETURN_NOT_OK(output->init_var_size(
ChunkedBuffer::BufferAddressing::CONTIGUOUS, std::move(chunk_sizes)));

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Making a note that we now need to clean up ChunkedBuffer as we don't need two different addressing types - they should all be CONTIGUOUS.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed, I will filed a follow up item for this for next sprint.

tiledb/sm/filter/filter_pipeline.cc Outdated Show resolved Hide resolved
Copy link
Member

@stavrospapadopoulos stavrospapadopoulos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually no backward compatibility issue, we should be good.

@KiterLuc KiterLuc force-pushed the lr/remove-selective-unfiltering/ch8874 branch from d5ffd4d to 9f71d2d Compare July 24, 2021 21:27
@KiterLuc KiterLuc closed this Jul 25, 2021
@KiterLuc KiterLuc force-pushed the lr/remove-selective-unfiltering/ch8874 branch from 9f71d2d to c592d0b Compare July 25, 2021 17:43
From what we know about our customer's use case, they either will read
the full array where this gains nothing (and actually this affects the
performance negatively because of the added code complexity), or very
targeted ones, for which the gains would be very small, unless the tile
capacity/extent is unusually large if the users have a poorly configured
array.

Also, this will allow us to implement compression codec for video and
imaging, which won't work with selective unfiltering.

---
TYPE: IMPROVEMENT
DESC: Remove selective unfiltering.
@KiterLuc KiterLuc reopened this Jul 25, 2021
@stavrospapadopoulos stavrospapadopoulos merged commit add102f into dev Jul 25, 2021
@stavrospapadopoulos stavrospapadopoulos deleted the lr/remove-selective-unfiltering/ch8874 branch July 25, 2021 18:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants