[SPARK-58609][SQL] Add archivePathFilter option to select inner archive entries

[akshatshenoi-db]

What changes were proposed in this pull request?

This PR adds a new file-source option archivePathFilter: a glob that selects which inner entries of an archive are ingested, matched against each entry's full path within the archive, so patterns like subdir/* and */*.csv work.

It continues the archive-read series (SPARK-57135 / SPARK-57321 CSV, SPARK-57419 JSON, SPARK-57478 text, SPARK-57479 XML, SPARK-57481 Avro, SPARK-57590 Parquet, SPARK-57591 ORC, SPARK-58382 binaryFile, SPARK-58110 trait extraction).

Details:

• FileSourceOptions gains the archivePathFilter option (the glob string) plus archivePathFilterPattern, a @transient lazy val holding the compiled Hadoop GlobPattern. That caches per FileSourceOptions instance, so the reads sharing one options object reuse a single matcher instead of recompiling per archive; it is transient because GlobPattern is not serializable, so the string travels and an executor recompiles on first use. Two kinds of call site cannot reach that value and carry the glob string instead: the schema-inference RDDs, which would otherwise have to capture the matcher in a closure, compile once per partition via mapPartitions; and the parallel footer/schema readers for Parquet and ORC, whose signatures are fixed by SchemaMergeUtils, read the glob from the Hadoop configuration. The glob string is validated on the driver, so an invalid glob raises a clear IllegalArgumentException naming the option.
• SupportsArchiveFormat.readArchiveEntries and the random-access readLocalizedEntries / localizeEntries take an optional filter (default None, so existing callers are unaffected), applied in shouldSkipEntry.
• The option is threaded from the read paths of CSV, JSON, text, XML, Avro, binaryFile, Parquet and ORC.

The filter is applied in addition to ignoredPathSegmentRegex (both must pass), so hidden entries stay hidden even when they match the glob.

Why are the changes needed?

pathGlobFilter cannot express this. It is applied during file listing, so it must already match the archive file itself for that archive to be read at all; it cannot simultaneously select a subset of the entries inside the archive. Without archivePathFilter there is no way to read only part of an archive, so a user wanting a few entries has to read and discard the rest.

Does this PR introduce any user-facing change?

Yes. A new read option archivePathFilter (a glob string, no default) is available to the file sources that support archive reads. When set, only archive entries whose full inner path matches the glob are ingested. Reads that do not set it are unaffected, and the whole archive-read feature remains gated by spark.sql.files.archive.reader.enabled (default false).

How was this patch tested?

New shared cases in ArchiveReadSuiteBase, which run for every (format, container) pair -- CSV, JSON, XML, text, Avro, Parquet, ORC and binaryFile across tar, zip and 7z:

• selection by inner path (sub/*),
• an extension glob matching across subdirectories, including an entry that does not match the glob,
• a glob matching no entry yields zero rows,
• composition with ignoredPathSegmentRegex (a hidden entry matching the glob is still skipped).

New unit cases in SupportsArchiveFormatSuite cover the entry-skip logic directly (glob matching the full path, * crossing /, no-match, interaction with hidden-entry filtering) and invalid-glob rejection.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code

[cloud-fan]

0 blocking, 1 non-blocking, 1 nit.
The filtering behavior and cross-format propagation are coherent, with two non-blocking cleanup items around matcher cache scope and repeated inference-time compilation.

Nits: 1 minor item (see inline comments).

Suggestions (1)

• sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala:185: This compilation runs once per archive in the flatMap callback, so partitions containing multiple archives repeatedly compile the same glob; JSON and XML inference have the same shape. Please compile once in mapPartitions and reuse the matcher across that partition's archive streams. -- see inline

Verification

I traced the option from FileSourceOptions through the shared archive-entry predicate and through scan and schema-inference paths for the supported formats. The filter is applied before parsing and composes with the existing per-segment hidden-path check; Parquet and ORC transport the string through Hadoop configuration for parallel schema work, while CSV, JSON, and XML currently compile it inside each archive callback.

PR metadata suggestions

• Correct the claim that archivePathFilterPattern is compiled once per executor JVM; it is cached per FileSourceOptions instance, and CSV/JSON/XML inference currently compiles once per archive.

[cloud-fan]

2 addressed, 0 remaining, 3 new. (3 newly introduced, 0 late catches, 0 previously raised.)
0 blocking, 2 non-blocking, 1 nit.
The filtering behavior is coherent across formats and the previous review concerns are addressed; three small cleanup items remain.

Design / architecture (1)

• .isaac/config.json:2: Remove .isaac/config.json; it is unrelated local tool state. -- see inline

Suggestions (1)

• sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala:180: Initialize the inference matcher lazily so ordinary-file partitions do not compile an unused glob. -- see inline

Nits: 1 minor item (see inline comments).

Verification

I traced the option from parsing and validation through the shared entry predicate, streaming and random-access scans, and each schema-inference transport path. The shared tests cover the user-visible selection contract and its composition with existing hidden-entry filtering.

[cloud-fan]

3 addressed, 0 remaining, 0 new.
0 blocking, 0 non-blocking, 0 nits.
The three concerns from the prior review are addressed, and the current implementation is coherent across scan and inference paths.

Verification

I traced option parsing and validation through the shared archive-entry predicate, streaming and localized readers, and each format-specific schema-inference transport path. The latest changes make the documented cache scope precise and use partition-local lazy matcher compilation for CSV, JSON, and XML inference.

PR metadata suggestions

• Correct the claim that the SupportsArchiveFormat helper parameters default to None; the current helper signatures require callers to pass the optional matcher explicitly.

[cloud-fan]

@akshatshenoi-db can you make the CI green?

[cloud-fan]

Merge Summary:

• merged into master 7d5a454
• merged into branch-4.x 59b7131
• merged into branch-4.3 fc8f161

Posted by merge_spark_pr.py