Skip to content

Initialize empty MDT file groups with native log files #19311

Description

@cshuo

Bug Description

What happened:

HoodieBackedTableMetadataWriter#initializeFileGroups initializes every metadata table file group by writing an empty HoodieDeleteBlock in the legacy inline log format. When native log format is enabled by default, MDT expects native .log.hfile files, so an empty file group may not be discovered by the file-system view before it receives records.

What you expected:

When native log format is enabled, initialize each empty MDT file group with a zero-record native .log.hfile containing the instant time, writer schema, and native log version in footer metadata. Continue using the empty inline delete block for legacy log format.

Steps to reproduce:

  1. Enable the metadata table with a table version that writes native logs.
  2. Initialize an MDT partition with file groups that receive no records during bootstrap.
  3. Load the metadata table file-system view.
  4. Observe that file groups without a native log file are not discovered.

Environment

Hudi version: current master
Query engine: engine-independent metadata table initialization
Relevant configs: native log format enabled

Related Issues

Parent feature issue: #14310

Metadata

Metadata

Assignees

Labels

type:bugBug reports and fixes

Type

No type

Fields

No fields configured for issues without a type.

Projects

Status
In progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions