Skip to content

HoodieTableState is write-only except latestCommitTimestamp; collapse it into a plain parameter #19465

Description

@voonhous

Problem

After #19458, latestCommitTimestamp is the only field of HoodieTableState (hudi-spark-common, HoodieBaseRelation.scala) that anything still reads -- two .withLatestCommitTime(tableState.latestCommitTimestamp.orNull) calls in HoodieMergeOnReadRDDV2.compute.

The other seven fields (tablePath, recordKeyField, orderingFields, usesVirtualKeys, metadataConfig, recordMergeImplClasses, recordMergeStrategyId) are write-only: populated once in HoodieBaseRelation.tableState and never read. The reads died in #17457 (FileGroupReader migration for all Spark datasource reads), #13544 and #12981; usesVirtualKeys appears never to have been read at all. The whole object (including a full HoodieMetadataConfig with its Properties) is a non-transient field of HoodieMergeOnReadRDDV2, so it is serialized into every MOR task for the sake of one Option[String].

Proposal

Delete HoodieTableState entirely and pass latestCommitTimestamp directly to HoodieMergeOnReadRDDV2:

  • HoodieBaseRelation: drop the HoodieTableState case class and the tableState lazy val (the recordMergerImpls local and the fileIndex.getMetadataConfig call there become dead too; the relation-level recordKeyField / orderingFields lazy vals must stay -- they are still read via mandatoryFieldsForMerging and the incremental relation traits)
  • HoodieMergeOnReadRDDV2: replace the tableState: HoodieTableState constructor parameter with latestCommitTimestamp: Option[String]
  • Update the three composeRDD call sites (MergeOnReadSnapshotRelation, MergeOnReadIncrementalRelationV1/V2)

Flagged by @wombatu-kun while reviewing #19458 ("worth dropping the other seven by the same criterion, or noting that they are deliberately kept - follow-up, not a blocker"); kept out of that PR to avoid growing an already-reviewed diff.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions