Problem
After #19458, latestCommitTimestamp is the only field of HoodieTableState (hudi-spark-common, HoodieBaseRelation.scala) that anything still reads -- two .withLatestCommitTime(tableState.latestCommitTimestamp.orNull) calls in HoodieMergeOnReadRDDV2.compute.
The other seven fields (tablePath, recordKeyField, orderingFields, usesVirtualKeys, metadataConfig, recordMergeImplClasses, recordMergeStrategyId) are write-only: populated once in HoodieBaseRelation.tableState and never read. The reads died in #17457 (FileGroupReader migration for all Spark datasource reads), #13544 and #12981; usesVirtualKeys appears never to have been read at all. The whole object (including a full HoodieMetadataConfig with its Properties) is a non-transient field of HoodieMergeOnReadRDDV2, so it is serialized into every MOR task for the sake of one Option[String].
Proposal
Delete HoodieTableState entirely and pass latestCommitTimestamp directly to HoodieMergeOnReadRDDV2:
HoodieBaseRelation: drop the HoodieTableState case class and the tableState lazy val (the recordMergerImpls local and the fileIndex.getMetadataConfig call there become dead too; the relation-level recordKeyField / orderingFields lazy vals must stay -- they are still read via mandatoryFieldsForMerging and the incremental relation traits)
HoodieMergeOnReadRDDV2: replace the tableState: HoodieTableState constructor parameter with latestCommitTimestamp: Option[String]
- Update the three
composeRDD call sites (MergeOnReadSnapshotRelation, MergeOnReadIncrementalRelationV1/V2)
Flagged by @wombatu-kun while reviewing #19458 ("worth dropping the other seven by the same criterion, or noting that they are deliberately kept - follow-up, not a blocker"); kept out of that PR to avoid growing an already-reviewed diff.
Problem
After #19458,
latestCommitTimestampis the only field ofHoodieTableState(hudi-spark-common,HoodieBaseRelation.scala) that anything still reads -- two.withLatestCommitTime(tableState.latestCommitTimestamp.orNull)calls inHoodieMergeOnReadRDDV2.compute.The other seven fields (
tablePath,recordKeyField,orderingFields,usesVirtualKeys,metadataConfig,recordMergeImplClasses,recordMergeStrategyId) are write-only: populated once inHoodieBaseRelation.tableStateand never read. The reads died in #17457 (FileGroupReader migration for all Spark datasource reads), #13544 and #12981;usesVirtualKeysappears never to have been read at all. The whole object (including a fullHoodieMetadataConfigwith itsProperties) is a non-transient field ofHoodieMergeOnReadRDDV2, so it is serialized into every MOR task for the sake of oneOption[String].Proposal
Delete
HoodieTableStateentirely and passlatestCommitTimestampdirectly toHoodieMergeOnReadRDDV2:HoodieBaseRelation: drop theHoodieTableStatecase class and thetableStatelazy val (therecordMergerImplslocal and thefileIndex.getMetadataConfigcall there become dead too; the relation-levelrecordKeyField/orderingFieldslazy vals must stay -- they are still read viamandatoryFieldsForMergingand the incremental relation traits)HoodieMergeOnReadRDDV2: replace thetableState: HoodieTableStateconstructor parameter withlatestCommitTimestamp: Option[String]composeRDDcall sites (MergeOnReadSnapshotRelation,MergeOnReadIncrementalRelationV1/V2)Flagged by @wombatu-kun while reviewing #19458 ("worth dropping the other seven by the same criterion, or noting that they are deliberately kept - follow-up, not a blocker"); kept out of that PR to avoid growing an already-reviewed diff.