Skip to content

Short circuit expensive comparissions in GroupColumn trait #23342

Description

@Rich-T-kid

Is your feature request related to a problem or challenge?

In GroupValuesColumn, equality between rows is checked by passing a shared boolean buffer to each column's vectorized_equal_to implementation. Each trait implementation sets a bit to true if the element already exists (matches) and leaves it false otherwise:

  for (col_idx, group_col) in self.group_values.iter().enumerate() {
            group_col.vectorized_equal_to(
                &self.vectorized_operation_buffers.equal_to_group_indices,
                &cols[col_idx],
                &self.vectorized_operation_buffers.equal_to_row_indices,
                &mut equal_to_results,
            );
        }

Since a row is only considered a match if every column reports equality, most implementations follows a short-circuiting pattern:

{
            if !equal_to_results.get_bit(idx) {
                continue;
            }

If an earlier column already cleared the bit for a given slot, later columns skip the (potentially expensive) comparison for that slot.

This short-circuiting is effective, but the columns are currently compared in input schema order, which is arbitrary with respect to comparison cost. As a result, we may end up running expensive comparisons before cheaper ones that could have ruled out the row earlier.

Describe the solution you'd like

Reorder the column comparisons so that cheaper comparisons run first and more expensive comparisons run last. This way, rows that are never going to match are eliminated early using inexpensive checks, and costly comparisons are only performed on rows that have survived all the cheaper filters, avoiding unnecessary work on rows already known not to match.

Describe alternatives you've considered

n/a

Additional context

n/a

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Fields

    No fields configured for Feature.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions