Yaohua628 and dongjoon-hyun [SPARK-38767][SQL] Support ignoreCorruptFiles and `ignoreMissingFil…
a92ef00 Apr 12, 2022
[SPARK-38767][SQL] Support `ignoreCorruptFiles` and `ignoreMissingFil…
…es` in Data Source options

### What changes were proposed in this pull request?
Support `ignoreCorruptFiles` and `ignoreMissingFiles` in Data Source options for both V1 and V2 file-based data sources.
```
spark.read...option("ignoreCorruptFiles", "true")...
spark.read...option("ignoreMissingFiles", "true")...
```

### Why are the changes needed?
Improve UX

### Does this PR introduce _any_ user-facing change?
Yes.

Previously:
```
spark.sql("set spark.sql.files.ignoreCorruptFiles=true")
spark.sql("set spark.sql.files.ignoreMissingFiles=true")
spark.read...
spark.read...
```

Now:
```
spark.read...option("ignoreCorruptFiles", "true")...
spark.read...option("ignoreMissingFiles", "true")...
```

### How was this patch tested?
Enhance existing UTs for ignoreMissingFiles + ignoreCorruptFiles

Closes #36069 from Yaohua628/spark-38767.

Authored-by: yaohua <yaohua.zhao@databricks.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
a92ef00