[SPARK-48649][SQL] Add "ignoreInvalidPartitionPaths" and "spark.sql.files.ignoreInvalidPartitionPaths" configs to allow ignoring invalid partition paths

[sadikovi]

What changes were proposed in this pull request?

This PR adds a new data source config ignoreInvalidPartitionPaths and SQL session configuration flag spark.sql.files.ignoreInvalidPartitionPaths to control the behaviour of skipping invalid partition paths (base paths).

When the config is enabled, it allows skipping invalid paths such as:

table/
  invalid/...
  part=1/...
  part=2/... 
  part=3/...

In this case, table/invalid path will be ignored.

Data source option takes precedence over the SQL config so with the code:

spark.conf.set("spark.sql.files.ignoreInvalidPartitionPaths", "false")

spark.read.format("parquet").option("ignoreInvalidPartitionPaths", "true").load(...)

the query would ignore invalid partitions, i.e. the flag will be enabled.

The config is disabled by default.

Why are the changes needed?

Allows ignoring invalid partition paths that cannot be parsed.

Does this PR introduce any user-facing change?

No. The added configs are disabled by default to have the exact same behaviour as before.

How was this patch tested?

I added a unit test for this.

Was this patch authored or co-authored using generative AI tooling?

No.

[sadikovi]

cc @cloud-fan @dongjoon-hyun @gengliangwang for review. Thank you.

[cloud-fan]

thanks, merging to master!

[sadikovi]

Thanks @cloud-fan. I forgot to ask, do we need any documentation updates or migration/release note for this?

[cloud-fan]

This is not a breaking change (it's a new feature), so migration guide is not needed.