Skip to content

Use state transition times for SimpleConcurrentFileWritesConflictResolutionStrategy and other enhancements. #16015

@hudi-bot

Description

@hudi-bot

Couple of enhancements we can make to Multiwriter conflict resolution strategies.

  • Similar to how git rebase works, writers when creating a requested commit should track the baseInstant on which they are executing, this will be the last completed commit. Note, last completed commit can have commit timestamp less than other completed commits in the timeline. Here, baseInstantTime is based on the state transition times which is the last commit written to the timeline. Conflict resolution strategies should happen based on this commit time. We can also avoid the race condition by acquiring lock during requested file creation.
  • Similar to IngestionPrimaryWriterBasedConflictResolutionStrategy class, we should use state transition times to find the conflicts. By using state transition times the conflict resolution works similar to git rebase.

JIRA info

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions