[HUDI-6445] Making some of Spark DS tests as functional

[nsivabalan]

Change Logs

Making some of Spark DS tests as functional.

spark-ds module (Job 3) is reaching the 3 hour limit in our azure runs. From test run time standpoint, here are the top tests

2023-05-26T09:45:58.2884790Z 56.932 org.apache.hudi.functional.TestIncrementalReadWithFullTableScan testFailEarlyForIncrViewQueryForNonExistingFiles{HoodieTableType}[2]
2023-05-26T09:45:58.2906456Z 53.483 org.apache.hudi.functional.TestIncrementalReadWithFullTableScan testFailEarlyForIncrViewQueryForNonExistingFiles{HoodieTableType}[1]
2023-05-26T09:45:58.2907521Z 53.222 org.apache.hudi.functional.cdc.TestCDCDataFrameSuite testMORDataSourceWrite{HoodieCDCSupplementalLoggingMode}[1]
2023-05-26T09:45:58.2907989Z 51.41 org.apache.hudi.functional.cdc.TestCDCDataFrameSuite testMORDataSourceWrite{HoodieCDCSupplementalLoggingMode}[3]
2023-05-26T09:45:58.2908437Z 51.036 org.apache.hudi.functional.cdc.TestCDCDataFrameSuite testMORDataSourceWrite{HoodieCDCSupplementalLoggingMode}[2]
2023-05-26T09:45:58.2908872Z 47.381 org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType, HoodieRecordType, String}[2]
2023-05-26T09:45:58.2909269Z 46.756 org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType, HoodieRecordType, String}[4]
2023-05-26T09:45:58.2909683Z 46.067 org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType, HoodieRecordType, String}[3]
2023-05-26T09:45:58.2910109Z 45.996 org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType, HoodieRecordType, String}[1]
2023-05-26T09:45:58.2910533Z 43.688 org.apache.hudi.functional.cdc.TestCDCDataFrameSuite testCOWDataSourceWrite{HoodieCDCSupplementalLoggingMode}[1]
2023-05-26T09:45:58.2911130Z 41.124 org.apache.hudi.functional.cdc.TestCDCDataFrameSuite testCOWDataSourceWrite{HoodieCDCSupplementalLoggingMode}[2]
2023-05-26T09:45:58.2911586Z 40.636 org.apache.hudi.functional.cdc.TestCDCDataFrameSuite testCOWDataSourceWrite{HoodieCDCSupplementalLoggingMode}[3]
2023-05-26T09:45:58.2912011Z 40.511 org.apache.hudi.functional.TestMORDataSourceWithBucketIndex testCountWithBucketIndex
2023-05-26T09:45:58.2912435Z 32.054 org.apache.hudi.functional.TestCOWDataSource testSparkPartitionByWithCustomKeyGenerator{HoodieRecordType}[1]

Looks like CDC tests are not tagged correctly as functional. so, fixing those there so that these will be run w/ Job4 instead of job3, thus reducing the total run time for job 3 atleast by 10 mins. 17 tests will move from Job3 to Job4.

Impact

Making some of Spark DS tests as functional

Risk level (write none, low medium or high below)

low

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change

• The config description must be updated if new configs are added or the default value of the configs are changed
• Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
  ticket number here and follow the instruction to make
  changes to the website.

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[nsivabalan]

@hudi-bot run azure

[hudi-bot]

CI report:

• d28ff94 UNKNOWN
• 43f5c3b Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[yihua]

This is no longer needed as we did the CI rebalancing in #10805.