[C++] Pushdown filters on augmented columns like fragment filename

[asfimport]

In the discussion on ARROW-15260, if we run the following code in R, we might expect it to push down the filter so we can just read in the relevant files:

  filter = Expression$create(
    "match_substring",
    Expression$field_ref("__filename"),
    options = list(pattern = "cyl=8")
  )

As mentioned by @westonpace:

"You might think we would get the hint and only read files matching that pattern. This is not the case. We will read the entire dataset and apply the "cyl=8" filter in memory.

If we want to pushdown filters on the filename column we will need to add some special logic."

Reporter: Nicola Crane / @thisisnic

Related issues:

• [R] open_dataset - add file_name as column (is related to)

_{Note: This issue was originally created as ARROW-16164. Please see the migration documentation for further details.}

[asfimport]

Weston Pace / @westonpace:
So this is possible. And something like regex on filename might be interesting. However, I'm not terribly motivated to work on this because:

• In the above example the user could establish a partitioning on cyl and then just filter for cyl == 8.

• For more general filename filtering the user can often do this themselves by creating a dataset, getting the list of files, picking the files they want, and then creating a new dataset from the smaller list of files.

  So it might be nice to first know of some key use cases that aren't solvable with other features.