From 6ee513d8556219398a22fb55ac76a231896633a3 Mon Sep 17 00:00:00 2001 From: chenhang Date: Sat, 4 May 2024 23:03:49 +0800 Subject: [PATCH 1/4] PIP-348: Trigger offload on topic load stage --- pip/pip-348.md | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100644 pip/pip-348.md diff --git a/pip/pip-348.md b/pip/pip-348.md new file mode 100644 index 0000000000000..9186804304b01 --- /dev/null +++ b/pip/pip-348.md @@ -0,0 +1,34 @@ +# PIP-348: Trigger offload on topic load stage + +# Background knowledge + +Pulsar tiered storage is introduced by [PIP-17](https://github.com/apache/pulsar/wiki/PIP-17:-Tiered-storage-for-Pulsar-topics) to offload cold data from BookKeeper to external storage. Ledger is the basic offload unit, and one ledger will trigger offload only when the ledger rollover. Pulsar topic offload can be triggered by the following ways: +- Manually trigger offload by using the `bin/pulsar-admin` command. +- Automatically trigger offload by the offload policy. + + +# Motivation +For triggering offload, the offload policy is the most common way. The offload policy can be defined in cluster level, namespace level and topic level, and the offload policy is triggered by the following ways: +- One ledger is closed or rollover +- Check the offload policy +- Trigger offload if the offload policy is satisfied + +If one topic has multiple ledgers and the latest ledgers rollover triggered offload, all the previous ledgers will be added into pending offload queue and trigger offload one by one. However, if the topic is unloaded and loaded again, the offload process will be interrupted and needs to waiting for the next ledger rollover to trigger offload. This will cause the offload process is not efficient and the offload process is not triggered in time. + + +# Goals + +## In Scope + +Trigger offload on topic load stage to improve the offload process efficiency and make sure the offload process is triggered in time. + + +# Detailed Design + +## Design & Implementation Details + +When the topic is loaded, we can check the offload policy to see if the offload policy is satisfied. If the offload policy is satisfied, we can trigger offload immediately. This will improve the offload process efficiency and make sure the offload process is triggered in time. + +# Links +* Mailing List discussion thread: +* Mailing List voting thread: From 4e8b66a2cabe57aa5b1f27de2522d5ad051f7a1c Mon Sep 17 00:00:00 2001 From: chenhang Date: Sat, 4 May 2024 23:06:57 +0800 Subject: [PATCH 2/4] update document --- pip/pip-348.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/pip/pip-348.md b/pip/pip-348.md index 9186804304b01..cc247e1090ff8 100644 --- a/pip/pip-348.md +++ b/pip/pip-348.md @@ -29,6 +29,11 @@ Trigger offload on topic load stage to improve the offload process efficiency an When the topic is loaded, we can check the offload policy to see if the offload policy is satisfied. If the offload policy is satisfied, we can trigger offload immediately. This will improve the offload process efficiency and make sure the offload process is triggered in time. + +# Backward & Forward Compatibility + +Fully compatible. + # Links * Mailing List discussion thread: * Mailing List voting thread: From 7056a83c1d0c76db9d6a02697910c5a0447dc7e0 Mon Sep 17 00:00:00 2001 From: chenhang Date: Mon, 6 May 2024 22:32:10 +0800 Subject: [PATCH 3/4] Update doc --- pip/pip-348.md | 1 + 1 file changed, 1 insertion(+) diff --git a/pip/pip-348.md b/pip/pip-348.md index cc247e1090ff8..6b1e2d8a55637 100644 --- a/pip/pip-348.md +++ b/pip/pip-348.md @@ -29,6 +29,7 @@ Trigger offload on topic load stage to improve the offload process efficiency an When the topic is loaded, we can check the offload policy to see if the offload policy is satisfied. If the offload policy is satisfied, we can trigger offload immediately. This will improve the offload process efficiency and make sure the offload process is triggered in time. +In order to reduce the impact on topic load when Pulsar is upgraded from the old versions, I introduce a flag named `triggerOffloadOnTopicLoad` to control whether enable this feature or not. # Backward & Forward Compatibility From 5ef2516cf8d68b6fe174312053f1fbeda7771679 Mon Sep 17 00:00:00 2001 From: chenhang Date: Thu, 9 May 2024 18:08:41 +0800 Subject: [PATCH 4/4] Update vote thread --- pip/pip-348.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pip/pip-348.md b/pip/pip-348.md index 6b1e2d8a55637..7661ef3685867 100644 --- a/pip/pip-348.md +++ b/pip/pip-348.md @@ -36,5 +36,5 @@ In order to reduce the impact on topic load when Pulsar is upgraded from the old Fully compatible. # Links -* Mailing List discussion thread: -* Mailing List voting thread: +* Mailing List discussion thread: https://lists.apache.org/thread/2ndomp8v4wkcykzthhlyjqfmswor88kv +* Mailing List voting thread: https://lists.apache.org/thread/q4mfn8x69hbgv19nmqx4dmknl3vsn9y8