Skip to content

HDDS-15562. EventNotification: persistence strategy for consumer checkpoint (via distributed RocksDB update)#10689

Open
gardenia wants to merge 1 commit into
apache:HDDS-13513_Event_Notification_FeatureBranchfrom
gardenia:HDDS-15562-DB-Checkpoint
Open

HDDS-15562. EventNotification: persistence strategy for consumer checkpoint (via distributed RocksDB update)#10689
gardenia wants to merge 1 commit into
apache:HDDS-13513_Event_Notification_FeatureBranchfrom
gardenia:HDDS-15562-DB-Checkpoint

Conversation

@gardenia

@gardenia gardenia commented Jul 8, 2026

Copy link
Copy Markdown

Please describe your PR in detail:

Persist the checkpoint within the OM RocksDB metadata table using a distributed Ratis update message so that another OM can pick up if the leader changes.

Technical Solution:

  • Update the RocksDB metadata table (OMSetEventNotificationCheckpointRequest) and distribute this to other OMs via Ratis (SetEventNotificationCheckpointRequest)

Checkpoint Throttling:

  • We update our local progress in-memory on every transaction batch, but throttle the persistent Ratis writes (configurable via OZONE_OM_PLUGIN_CHECKPOINT_SAVE_INTERVAL, defaulting to every 100 transactions).
  • The intent is to prevent saturating the Ratis log or causing write storms on RocksDB while maintain our at-least-once delivery guarantee.
  • If a Ratis write fails, throttling is bypassed to ensure the next update cycle retries immediately.

Error Handling:

  • Both loading and saving failures are non-blocking for background polling progress
  • On load failures, we log a warning and fallback to starting from transaction 0/null.
  • On save failures, we log a warning but always advance progress in-memory to ensure progress but conform to at-least-once event delivery.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15562

How was this patch tested?

Unit tests

@gardenia gardenia changed the base branch from master to HDDS-13513_Event_Notification_FeatureBranch July 8, 2026 08:34
@gardenia

gardenia commented Jul 8, 2026

Copy link
Copy Markdown
Author

NOTE: this is a rework of the original implementation which used Ozone filesystem bucket property metadata to store the properties (PR and review comments from @ChenSammi can be found here #10515).

This approach of using a distributed write to the RocksDB metadata table for persisting the checkpoint was suggested by @ChenSammi in review comments you can find on the above PR.

Persist the checkpoint within the OM RocksDB metadata table using a distributed
Ratis update message so that another OM can pick up if the leader changes.

Technical Solution:
* Update the RocksDB metadata table (OMSetEventNotificationCheckpointRequest) and
distribute this to other OMs via Ratis (SetEventNotificationCheckpointRequest)

Checkpoint Throttling:
* We update our local progress in-memory on every transaction batch, but throttle
the persistent Ratis writes (configurable via OZONE_OM_PLUGIN_CHECKPOINT_SAVE_INTERVAL,
defaulting to every 100 transactions).
* The intent is to prevent saturating the Ratis log or causing write storms on RocksDB
while maintain our at-least-once delivery guarantee.
* If a Ratis write fails, throttling is bypassed to ensure the next
update cycle retries immediately.

Error Handling:
* Both loading and saving failures are non-blocking for background polling progress
* On load failures, we log a warning and fallback to starting from transaction 0/null.
* On save failures, we log a warning but always advance progress in-memory to ensure
progress but conform to at-least-once event delivery.

Detailed Changes:
- Protobuf:
  - Add SetEventNotificationCheckpoint type and request/response payloads to OmClientProtocol.proto.
- OmUtils & OzoneManagerRatisUtils:
  - Wire SetEventNotificationCheckpoint as a write request.
  - Route SetEventNotificationCheckpoint to its corresponding OMRequest class.
- Requests & Responses:
  - Create OMSetEventNotificationCheckpointRequest to write checkpoint values to metaTable.
  - Create OMSetEventNotificationCheckpointResponse to persist values in metaTable via addToDBBatch.
- Strategy:
  - Expose NotificationCheckpointStrategy interface with load, save, and reset.
  - Implement OzoneDbCheckpointStrategy to query and save checkpoints atomically via the new OM requests.
  - Added reset() call to durably clear checkpoints during self-healing.
- Plugin Context & Poller:
  - Wire the checkpoint strategy through OMEventListenerPluginContext and OMEventListenerPluginManager.
  - Integrate strategy into the poller and publisher. Both loading and saving failures are non-blocking,
  falling back to starting from 0/null on load failures and continuing in-memory progress on save failures.
@gardenia gardenia force-pushed the HDDS-15562-DB-Checkpoint branch from 8936a16 to 2ed2e10 Compare July 8, 2026 10:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant