offload bucket per namespace by KannarFr · Pull Request #4739 · apache/pulsar

KannarFr · 2019-07-16T15:16:48Z

I started to implement #4547.

sijie

It seems this PR only introduces the offload policies but it doesn't implement the logic to let the offloader using the offload polices. Is that done intentionally?

pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/Namespaces.java

pulsar-common/src/main/java/org/apache/pulsar/common/tieredStorage/OffloadType.java

pulsar-common/src/main/java/org/apache/pulsar/common/policies/data/OffloadPolicies.java

KannarFr

I removed OffloadType to String. And fixed other points.

KannarFr · 2019-07-17T21:18:43Z

I added the logic about retrieving offloadPolicies for each NamespaceName, and get a Map<NamespaceName, LedgerOffloadManager>, there are todo on FileSystem and TieredStorage way to override default tiered storage configuration when a custom for NamespaceName exists. Now I need to implment the logic to schedule and run the offload.

KannarFr · 2019-07-18T10:12:54Z

@sijie The goal is to work with your reviews between each implement choices to ensure I'm doing well.

sijie · 2019-07-18T16:46:44Z

@jiazhai can you please review this tiered storage related change?

pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java

managed-ledger/src/main/java/org/apache/bookkeeper/mledger/LedgerOffloaderFactory.java

pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java

jiazhai · 2019-07-29T12:04:34Z

It would be good to add some test for this change.

KannarFr · 2019-07-30T08:52:21Z

pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiOffloadTest.java

@jiazhai I removed it because of the new lazy load feature of offload policies even for non defined offloaders. WDYT about removing this?

KannarFr · 2019-08-05T11:05:54Z

@sijie @jiazhai can you review before going further?

sijie

@KannarFr overall looks good. However I didn't see how the offload policies used by actual offloader. do I miss anything?

pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java

KannarFr · 2019-08-12T12:25:40Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java

@sijie If I'm correct, it's done here.

But then I don't know which ledgerFactory should be used, not sure about what their job :/ can you explain them?

@jiazhai maybe? :D

@sijie @jiazhai poke

cdbartholomew · 2019-11-02T22:40:23Z

I think this is a great feature. Is there anything I can do to help out?

aahmed-se · 2019-11-02T22:41:46Z

@cdbartholomew rebase the pr on master and add more tests.

sijie · 2019-11-03T01:07:31Z

@aahmed-se I would suggest asking the original author first before taking over the PR.

KannarFr · 2019-11-03T03:56:43Z

Hey, was waiting for feedback about the way I already done some work. I will be pleased to have your help. Sure first thing is to rebase from master.

cdbartholomew · 2019-11-04T11:11:18Z

Hey, was waiting for feedback about the way I already done some work. I will be pleased to have your help. Sure first thing is to rebase from master.

Ok, I'll rebase from master and take a look at adding some tests.

sijie

@KannarFr the general direction of this change looks good to me.

KannarFr · 2019-11-26T11:04:47Z

@cdbartholomew Hi, do you still want to work on it? :)

cdbartholomew · 2020-01-08T21:55:42Z

@KannarFr sorry about taking so long to respond. I have been really busy and haven't been able to look at it. However, I still think this is a very useful change, and I think I have talked @zzzming into working on it. :-)

KannarFr · 2020-01-09T14:53:04Z

@cdbartholomew ok, I'm rebasing from master this PR.

…offloadPolicies parameter

jiazhai · 2020-01-18T13:11:50Z

retest this please

gaoran10 · 2020-01-18T13:35:14Z

@KannarFr Hi, this is a good feature, I want to use it in my project, do you want finish it ?

sijie · 2020-01-18T21:37:09Z

@gaoran10 I guess @KannarFr is not actively working on this at this moment. If there is a need, feel free to continue the work based on this pull request.

gaoran10 · 2020-01-19T03:47:32Z

@sijie @KannarFr I'm glad to complete the feature base on this pull request.

KannarFr · 2020-01-20T12:25:25Z

@gaoran10 Hi, go for it :).

KannarFr · 2020-01-31T14:31:26Z

@gaoran10 any news? :)

gaoran10 · 2020-01-31T21:11:08Z

@KannarFr Hi, I finish this feature at #6183, have a look please.

KannarFr · 2020-01-31T23:02:10Z

@gaoran10 Oh, you rewrite all commits? Anyway, thanks for the feature, can't wait to use it :p.

codelipenghui · 2020-02-10T12:49:43Z

@KannarFr The #6183 merged, I would close this PR.

sijie · 2020-02-13T23:57:46Z

Thanks @KannarFr for the contribution!

codelipenghui · 2020-05-19T05:56:59Z

Since this feature is released in 2.5.1, so I added a release/2.5.1 tag for it.

KannarFr force-pushed the tieredstorage-per-namespace branch from 1050eeb to b547171 Compare July 16, 2019 15:22

sijie requested changes Jul 17, 2019

View reviewed changes

KannarFr force-pushed the tieredstorage-per-namespace branch 2 times, most recently from c2f13c6 to 2696882 Compare July 17, 2019 11:52

KannarFr commented Jul 17, 2019

View reviewed changes

sijie requested a review from jiazhai July 18, 2019 16:46

sijie assigned sijie and KannarFr and unassigned sijie Jul 18, 2019

sijie added the area/tieredstorage label Jul 18, 2019

sijie added this to the 2.5.0 milestone Jul 18, 2019

sijie requested changes Jul 21, 2019

View reviewed changes

jiazhai reviewed Jul 29, 2019

View reviewed changes

pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java Outdated Show resolved Hide resolved

KannarFr force-pushed the tieredstorage-per-namespace branch from 258074f to 2e227e5 Compare July 30, 2019 08:45

KannarFr commented Jul 30, 2019

View reviewed changes

KannarFr force-pushed the tieredstorage-per-namespace branch from 2e227e5 to 7bbe81f Compare July 30, 2019 09:00

sijie reviewed Aug 10, 2019

View reviewed changes

pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java Outdated Show resolved Hide resolved

KannarFr force-pushed the tieredstorage-per-namespace branch from 7bbe81f to 522c3de Compare August 12, 2019 12:17

KannarFr commented Aug 12, 2019

View reviewed changes

sijie reviewed Nov 25, 2019

View reviewed changes

KannarFr added 10 commits January 9, 2020 17:08

add offload policies

9743e76

add internal offload policies setter for namespace

d3fa073

add offload per namespace method to pulsar-client-admin

6a677c9

set method naming more relevant

d565433

offloadtype is now string

9896db3

create offloaders per namespaceName on PulsarService

84556fb

make it generic: replace old offloaderfactory method by new one with …

6671cce

…offloadPolicies parameter

remove api v1 management

3250579

lazy load ledgeroffloaders

0aa1a37

remove star imports

06e4584

KannarFr force-pushed the tieredstorage-per-namespace branch from 522c3de to 06e4584 Compare January 9, 2020 16:23

fix

0f8e057

gaoran10 mentioned this pull request Feb 10, 2020

Namespace level offloader #6183

Merged

codelipenghui closed this Feb 10, 2020

KannarFr deleted the tieredstorage-per-namespace branch February 10, 2020 12:57

codelipenghui added the release/2.5.1 label May 19, 2020

Conversation

KannarFr commented Jul 16, 2019

Uh oh!

sijie left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

KannarFr left a comment

Choose a reason for hiding this comment

Uh oh!

KannarFr commented Jul 17, 2019

Uh oh!

KannarFr commented Jul 18, 2019

Uh oh!

sijie commented Jul 18, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jiazhai commented Jul 29, 2019

Uh oh!

KannarFr Jul 30, 2019

Choose a reason for hiding this comment

Uh oh!

KannarFr commented Aug 5, 2019

Uh oh!

sijie left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

KannarFr Aug 12, 2019

Choose a reason for hiding this comment

Uh oh!

KannarFr Aug 12, 2019

Choose a reason for hiding this comment

Uh oh!

KannarFr Aug 12, 2019

Choose a reason for hiding this comment

Uh oh!

KannarFr Aug 26, 2019

Choose a reason for hiding this comment

Uh oh!

cdbartholomew commented Nov 2, 2019

Uh oh!

aahmed-se commented Nov 2, 2019

Uh oh!

sijie commented Nov 3, 2019

Uh oh!

KannarFr commented Nov 3, 2019

Uh oh!

cdbartholomew commented Nov 4, 2019

Uh oh!

sijie left a comment

Choose a reason for hiding this comment

Uh oh!

KannarFr commented Nov 26, 2019

Uh oh!

cdbartholomew commented Jan 8, 2020

Uh oh!

KannarFr commented Jan 9, 2020

Uh oh!

jiazhai commented Jan 18, 2020

Uh oh!

gaoran10 commented Jan 18, 2020

Uh oh!

sijie commented Jan 18, 2020

Uh oh!

gaoran10 commented Jan 19, 2020

Uh oh!

KannarFr commented Jan 20, 2020

Uh oh!

KannarFr commented Jan 31, 2020

Uh oh!

gaoran10 commented Jan 31, 2020

Uh oh!

KannarFr commented Jan 31, 2020