Skip to content

Report metrics on the number of duplicates after dedup #15668

Open
Open
Report metrics on the number of duplicates after dedup#15668
Labels
from-jirapriority:highSignificant impact; potential bugstype:improvementImprovements to existing functionality
@hudi-bot

Description

@hudi-bot
hudi-bot
opened on Nov 30, 2025

In Delastreamer, user can set --filter-dupes to dedup the input records.  Currently, Hudi does not report any metric on the duplicates.  It would be good to report such metrics and monitor how many records are filtered in each commit.

JIRA info

Metadata

Metadata

Assignees

No one assigned

    Labels

    from-jirapriority:highSignificant impact; potential bugstype:improvementImprovements to existing functionality

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions