Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improve][pip] PIP-350: Allow to disable the managedLedgerOffloadDeletionLagInMillis #22688

Merged
merged 3 commits into from
May 15, 2024
Merged

[improve][pip] PIP-350: Allow to disable the managedLedgerOffloadDeletionLagInMillis #22688

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a4d7375
[improve][pip] PIP-350: Allow to disable the managedLedgerOffloadDele…
zymap May 10, 2024
921e029
add the discussion link
zymap May 10, 2024
c5bc923
Update the voting thread
zymap May 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions pip/pip-350.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# PIP-350: Allow to disable the managedLedgerOffloadDeletionLagInMillis

# Background knowledge

https://pulsar.apache.org/docs/3.2.x/tiered-storage-overview/
Pulsar provides the ability to offload the data from bookkeeper to the cloud storage with the tiered storage.
Once the data is offloaded to the cloud storage, the data in the bookkeeper can be deleted after a certain period of time.
We use the managedLedgerOffloadDeletionLagInMillis to control the deletion lag time for the offloaded data.
The default value of managedLedgerOffloadDeletionLagInMillis is 4 hours. It means the offloaded data will be deleted after 4 hours by default.

# Motivation

In some test scenarios, we want to disable the deletionLag and never delete the data from the bookkeeper.
Then when the tiered storage data is broken, we can still read the data from the bookkeeper.

# Goals

## In Scope

Never deletes the bookkeeper data when the managedLedgerOffloadDeletionLagInMillis is set to -1.

# Detailed Design

## Design & Implementation Details

Only need to check the value of managedLedgerOffloadDeletionLagInMillis in the ManagedLedgerImpl when it is going to delete the bookkeeper data.
https://github.com/apache/pulsar/blob/774a5d42e8342ee50395cf3626b9e7af27da849e/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L2579

# Backward & Forward Compatibility

Fully compatible.

# Links

* Mailing List discussion thread: https://lists.apache.org/thread/7tlpkcm2933ddg95kgrb42943r4gq3v9
* Mailing List voting thread: https://lists.apache.org/thread/c3rh530dlwo6nhrdflpw0mjck85hhfbx