[HUDI-5261] Use Proper Parallelism for Engine Context APIs

[jonvex]

Change Logs

A lot of occurrences are using number of items as parallelism, which affect performance. Parallelism should be based on num cores available in the cluster and set by user via parallelism configs.

Impact

Better, more tunable performance.

Risk level (write none, low medium or high below)

low

Documentation Update

N/A

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[jonvex]

FileSystemBackedTableMetadata has config
DEFAULT_LISTING_PARALLELISM = 1500;
which seems pretty high

[jonvex]

TimelineServerPerf has numExecuters with a default of 10

But then also has numCoresPerExecutor also with a default of 10

Something seems off here. Maybe it's supposed to be numExecutors per core? Whatever it is, those configs seem to conflict

[hudi-bot]

CI report:

• 46cb8c2 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build