Skip to content

fix: balance list_contains OR tree for large IN-list filters#37

Merged
phillipleblanc merged 1 commit into
spiceai-52from
phillip/260407-balance-list-contains-or
Apr 7, 2026
Merged

fix: balance list_contains OR tree for large IN-list filters#37
phillipleblanc merged 1 commit into
spiceai-52from
phillip/260407-balance-list-contains-or

Conversation

@phillipleblanc
Copy link
Copy Markdown

Summary

  • build constant-list list_contains OR combinations as a balanced tree instead of a left-deep chain
  • add a regression test for large constant-list evaluation
  • add a structural test to verify balanced OR depth

Testing

  • cargo test -p vortex-array scalar_fn::fns::list_contains -- --nocapture
  • validated locally from spiceai/spiceai by overriding Vortex to this checkout and rerunning the SF1 staging_table SpiceBench flow past the previous part/lineitem stack-overflow point

Copilot AI review requested due to automatic review settings April 7, 2026 09:09
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses stack overflows when evaluating list_contains against large constant IN-lists by building the OR-combination of element-wise equality checks as a balanced tree rather than a left-deep chain, and adds regression/structural tests to prevent reintroducing the issue.

Changes:

  • Refactor constant-list list_contains evaluation to build a balanced OR reduction tree (or_arrays_balanced).
  • Add a regression test for large constant lists (2048 elements) to ensure evaluation completes correctly.
  • Add a structural test to validate the resulting OR tree depth is bounded.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread vortex-array/src/scalar_fn/fns/list_contains/mod.rs
@phillipleblanc phillipleblanc merged commit d694abd into spiceai-52 Apr 7, 2026
18 of 49 checks passed
@phillipleblanc phillipleblanc deleted the phillip/260407-balance-list-contains-or branch April 7, 2026 13:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants