Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve][pip] PIP-350: Allow to disable the managedLedgerOffloadDeletionLagInMillis #22688

Merged
merged 3 commits into from
May 15, 2024
Merged
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
36 changes: 36 additions & 0 deletions pip/pip-350.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# PIP-350: Allow to disable the managedLedgerOffloadDeletionLagInMillis

# Background knowledge

https://pulsar.apache.org/docs/3.2.x/tiered-storage-overview/
Pulsar provides the ability to offload the data from bookkeeper to the cloud storage with the tiered storage.
Once the data is offloaded to the cloud storage, the data in the bookkeeper can be deleted after a certain period of time.
We use the managedLedgerOffloadDeletionLagInMillis to control the deletion lag time for the offloaded data.
The default value of managedLedgerOffloadDeletionLagInMillis is 4 hours. It means the offloaded data will be deleted after 4 hours by default.

# Motivation

In some test scenarios, we want to disable the deletionLag and never delete the data from the bookkeeper.
Then when the tiered storage data is broken, we can still read the data from the bookkeeper.

# Goals

## In Scope

Never deletes the bookkeeper data when the managedLedgerOffloadDeletionLagInMillis is set to -1.

# Detailed Design

## Design & Implementation Details

Only need to check the value of managedLedgerOffloadDeletionLagInMillis in the ManagedLedgerImpl when it is going to delete the bookkeeper data.
https://github.com/apache/pulsar/blob/774a5d42e8342ee50395cf3626b9e7af27da849e/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L2579

# Backward & Forward Compatibility

Fully compatible.

# Links

* Mailing List discussion thread: https://lists.apache.org/thread/7tlpkcm2933ddg95kgrb42943r4gq3v9
* Mailing List voting thread: