Pass metadata extractors to FileScanRDD [databricks]

[razajafri]

This PR handles the change that was made in Spark. We are passing through the metdata extractors from the fileFormat to the FileScanRDD.

Changes

• Created a shim for 350+ to pass the metadata extractors

fixes #8766

[gerashegalov]

if we pass relation to access fileFormat to access fileConstantMetadataExtractors and only the latter is shim-specific should we just pass fileFormat as an option to getFileScanRDD?

[gerashegalov]

Typically in Scala calling .get on an Option is an anti-pattern.

Suggested change

	if (relation.isDefined) {
	new FileScanRDD(sparkSession, readFunction, filePartitions, readDataSchema, metadataColumns,
	metadataExtractors = relation.get.fileFormat.fileConstantMetadataExtractors)
	} else {
	new FileScanRDD(sparkSession, readFunction, filePartitions, readDataSchema, metadataColumns)
	}
	new FileScanRDD(sparkSession, readFunction, filePartitions, readDataSchema, metadataColumns,
	relation.map(_.fileFormat.fileConstantMetadataExtractors).getOrElse(Map.empty))

[gerashegalov]

if you move this arg to the last position and give it a default value None, you probably will have fewer lines to modify

[razajafri]

Thanks for the review and suggestions. PTAL again

[gerashegalov]

LGTM but need to update copyrights

can either use a pre-commit hook or invoke directly

export SPARK_RAPIDS_AUTO_COPYRIGHTER=ON 
git diff origin/branch-24.04..HEAD --name-status | \
  awk '/^M\s+/ { print $2}' | \
  xargs ./scripts/auto-copyrighter.sh

[gerashegalov]

Update copyright

[gerashegalov]

Update copyright year

[gerashegalov]

LGTM

[gerashegalov]

build