[HUDI-7073] Fix schema projection in file group reader-based parquet file format

[linliu-code]

Change Logs

As above. When turning on the file group reader-based parquet file format for MOR snapshot queries, the following exception is thrown in a few tests although ts exists in the schema

2023-11-09T21:08:42.7345689Z - Test Call show_logfile_metadata Procedure *** FAILED ***
2023-11-09T21:08:42.7347213Z   java.lang.IllegalArgumentException: Field ts does not exist in the data schema
2023-11-09T21:08:42.7349324Z   at org.apache.spark.sql.execution.datasources.parquet.HoodieFileGroupReaderBasedParquetFileFormat.$anonfun$generateRequiredSchemaWithMandatory$4(HoodieFileGroupReaderBasedParquetFileFormat.scala:277)
2023-11-09T21:08:42.7407685Z   at scala.Option.getOrElse(Option.scala:189)
2023-11-09T21:08:42.7409315Z   at org.apache.spark.sql.execution.datasources.parquet.HoodieFileGroupReaderBasedParquetFileFormat.$anonfun$generateRequiredSchemaWithMandatory$3(HoodieFileGroupReaderBasedParquetFileFormat.scala:277)

• To fix the schema project, we search fields in table schema. Previously we only search the data schema for a field, but
  it can also be contained in the partition schema. We add this logic.
• Update the manually created schema to match the one created through spark sql.

Impact

Bug fix.

Risk level

low

Documentation Update

N/A

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[hudi-bot]

CI report:

• edfa00f UNKNOWN
• a2e58e5 Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[yihua]

Tests pass based on #10044 which cherry-picks the change in this PR.