[Ruby] Multiple filter conditions in Arrow::Table.load

[collimarco]

Describe the enhancement requested

I have this code:

table = Arrow::Table.load(s3_uri, format: :parquet)
puts table.slice { |slicer| (slicer['status'] == 200) & (slicer['message'].match_substring? 'foo') }

Now I would like to rewrite it more efficiently using condition pushdown:

table = Arrow::Table.load(s3_uri, format: :parquet, filter: [[:equal, :status, 200], [:match_substring, :message, 'foo']])
puts table

However this code doesn't work (invalid argument Array for the filter).

Any idea how to rewrite it correctly? I can't find any documentation about multiple filter conditions.

Component(s)

Ruby

[kou]

Could you try filter: [:and, [:equal, :status, 200], [:match_substring, :message, 'foo']]?

[collimarco]

@kou It seems to work, but there is another problem with the function in the filter. Now I get this error:

/Users/collimarco/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/gobject-introspection-4.1.6/lib/gobject-introspection/loader.rb:705:in `invoke': 
[scanner-builder][filter][set]: Invalid: Function 'match_substring' accepts 1 arguments but 2 passed (Arrow::Error::Invalid)

[kou]

Ah, we need to create an option object for match_substring:

match_substring_options = Arrow::MatchSubstringOptions.new
match_substring_options.pattern = 'foo'
table = Arrow::Table.load(s3_uri, format: :parquet, filter: [:and, [:equal, :status, 200], [:match_substring, :message, match_substring_options]])

But filter: [:and, [:equal, :status, 200], [:match_substring, :message, {pattern: 'foo'}]] shortcut should be implemented.

[collimarco]

@kou Unfortunately the new code gives a segmentation fault:

searchparquet.rb: [BUG] Segmentation fault at 0x0000000000000000
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-darwin22]

[kou]

Oh, #35927 fixes it.