Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable predicate pushdown for categorical dimension filters #1227

Merged
merged 7 commits into from
Jun 25, 2024
Merged

Enable predicate pushdown for categorical dimension filters #1227

merged 7 commits into from
Jun 25, 2024

Commits on Jun 25, 2024

  1. Enable predicate pushdown for categorical dimension filters 

    We now have the ability to push down filter predicates within
the DataflowPlan. We start with categorical dimension filters,
as they are the simplest.

This change simply tracks the where filters applied at the measure
node and pushes all of them down to the construction of the source
node for evaluation. At this time a filter is eligible to be applied
to the source node if it only contains references to categorical dimensions
that originate from the same, singular semantic model definition that
feeds into the source node in question.

We do not support time dimensions at this time, as they can cause strange
interactions with things like cumulative metrics, which could result in
inappropriate input filtering that produces non-obviously censored metric
results.

We also do not support entities at this time, as entities may be defined
in multiple semantic models and as such filters must be applied with more
care to ensure we are correctly accounting for the entity link paths to
the relevant source node, if any, when we apply the filter.

The snapshot test changes for existing snapshots highlight the new behavior,
while the added test snapshots demonstrate specific circumstances of
interest.
    @tlento
    tlento committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    b4c5472 View commit details
    Browse the repository at this point in the history

  2. Add predicate pushdown rendering tests

    @tlento
    tlento committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    ef6fc82 View commit details
    Browse the repository at this point in the history

  3. Update SQL engine snapshots

    @tlento
    tlento committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    df90189 View commit details
    Browse the repository at this point in the history

  4. changelog

    @tlento
    tlento committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    f93031e View commit details
    Browse the repository at this point in the history

  5. Fix pushdown operations on joined in dimensions 

    Note for reviewers, as this will be squashed:

Pushdown on joined in dimensions was not working as expected. In
the original predicate pushdown rendering tests, most joined in
dimensions were being skipped for pushdown operations.

The root cause of the issue was the semantic model source value
we were accessing, which actually included the complete history
of all semantic model inputs for the joined in dimension.

Fixing that problem uncovered a separate issue, which is we were
inappropriately pushing filters down past outer join expressions.

This commit fixes both issues at once - we now only push down
on the "safe" side of an outer join (the left side for LEFT OUTER
and not at all for FULL OUTER joins), and we evaluate pushdown
based on the singuolar semantic model source where each element is
defined.
    @tlento
    tlento committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    4bc075e View commit details
    Browse the repository at this point in the history

  6. Update SQL Engine Snapshots again

    @tlento
    tlento committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    e9dcde3 View commit details
    Browse the repository at this point in the history

  7. Remove unnecessary continue

    @tlento
    tlento committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    ca3043c View commit details
    Browse the repository at this point in the history