[PIP-105] Part-2 Support pluggable entry filter in Dispatcher #12970

315157973 · 2021-11-25T04:51:28Z

Motivation

see #12269

Modifications

Add pluggable entry filter

Documentation

[ x ] doc-required

codelipenghui · 2021-11-25T07:22:46Z

conf/broker.conf

+# Class name of Pluggable entry filter that can decide whether the entry needs to be filtered
+# You can use this class to decide which entries can be sent to consumers.
+# Multiple classes need to be separated by commas.
+entryFilterClassNames=


Do we need a directory to maintain all the filters? otherwise, we need to copy the filter jar to the lib dir.

Another point is we should load the class by a separate classloader that extends from the pulsar classloader?

Anonymitaet

@315157973 I see you've added annotations in API files, except that, will you need descriptions (what the feature is and how to use this feature) to docs? Somewhere like here?

conf/broker.conf

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/EntryFilter.java

…ntryFilter.java Co-authored-by: Anonymitaet <50226895+Anonymitaet@users.noreply.github.com>

eolivelli

Great work.
Please take a look to my comments

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/EntryFilter.java

eolivelli · 2021-11-26T06:42:59Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/EntryFilter.java

+        private SubscriptionOption subscriptionOption;
+        private MessageMetadata msgMetadata;
+
+        public void reset() {


This method is only for the implementation, do not keep it here

I think Context does not need to be made into an interface, just make it into a class directly

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/EntryFilter.java

pulsar-broker/src/test/java/org/apache/pulsar/broker/service/plugin/EntryFilterForTest2.java

315157973 · 2021-11-26T07:28:38Z

Hello, I have processed as required, please take a look again @eolivelli

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/plugin/EntryFilterProvider.java

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractBaseDispatcher.java

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/plugin/FilterContext.java

315157973 · 2021-11-26T12:58:22Z

@codelipenghui @eolivelli PTAL

eolivelli

I did my last pass.
I am sorry I missed some comments on my first pass.

...broker/src/main/java/org/apache/pulsar/broker/service/plugin/EntryFilterWithClassLoader.java

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/plugin/EntryFilterProvider.java

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/plugin/EntryFilter.java

315157973 · 2021-11-27T01:16:05Z

@eolivelli All done, thanks

315157973 · 2021-11-29T02:43:22Z

@eolivelli PTAL

wolfstudy · 2021-11-30T02:59:31Z

ping @eolivelli PTAL thanks

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractBaseDispatcher.java

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/plugin/EntryFilterProvider.java

pulsar-broker/src/test/java/org/apache/pulsar/broker/service/plugin/EntryFilterTest.java

eolivelli

LGTM great work !

* up/master: (75 commits) [website][upgrade]feat: website upgrade / docs migration - 2.5.1 Get Started/Concepts and Architecture/Pulsar Schema (apache#13030) Fix environment variable assignment in startup scripts (apache#13025) update 2.8.x (apache#13029) [Doc] add tips for Pulsar tools (apache#13044) Suggest to use tlsPort instead of deprecated TlsEnable (apache#13039) Integration tests for function-worker rebalance and drain operations. (apache#13058) fix(functions): missing runtime set in GoInstanceConfig (apache#13031) [pulsar-admin] Add get-replicated-subscription-status command for topic (apache#12891) [Broker] Consider topics in pulsar/system namespace as system topics (apache#13050) Fix typo: correct sizeUint to sizeUnit (apache#13040) fix-12894 (apache#12896) Don't attempt to delete pending ack store unless transactions are enabled (apache#13041) [Perf] Evaluate the current protocol version once (apache#13045) Fix Issue apache#12885, Unordered consuming case in Key_Shared subscription (apache#12890) [broker]Optimize topicMaxMessageSize with topic local cache. (apache#12830) [PIP-105] Part-2 Support pluggable entry filter in Dispatcher (apache#12970) [website] Modify admin-api-topic.md document (apache#12996) add missed import (apache#13037) [metadata] Add RocksdbMetadataStore (apache#12776) [C] Add pulsar_client_subscribe_multi_topics and pulsar_client_subscribe_pattern (apache#12965) ... # Conflicts: # site2/website-next/docusaurus.config.js # site2/website-next/versioned_sidebars/version-2.6.1-sidebars.json # site2/website-next/versions.json

…#12970)

315157973 added 3 commits November 23, 2021 16:34

support entry filter

e301f88

Merge branch 'master' into plugin-filter

04f203a

Support pluggable entry filter in Dispatcher

3bac1be

315157973 requested review from hangc0276, eolivelli, codelipenghui and Anonymitaet November 25, 2021 04:51

315157973 self-assigned this Nov 25, 2021

github-actions bot added the doc-required Your PR changes impact docs and you will update later. label Nov 25, 2021

315157973 added 2 commits November 25, 2021 13:06

fix check style

b1b033a

Support multi entry filter

4dccf98

codelipenghui reviewed Nov 25, 2021

View reviewed changes

Anonymitaet reviewed Nov 25, 2021

View reviewed changes

315157973 and others added 7 commits November 25, 2021 17:20

Update pulsar-broker/src/main/java/org/apache/pulsar/broker/service/E…

a215b6b

…ntryFilter.java Co-authored-by: Anonymitaet <50226895+Anonymitaet@users.noreply.github.com>

Update pulsar-broker/src/main/java/org/apache/pulsar/broker/service/E…

942812f

…ntryFilter.java Co-authored-by: Anonymitaet <50226895+Anonymitaet@users.noreply.github.com>

Update pulsar-broker/src/main/java/org/apache/pulsar/broker/service/E…

e49760f

…ntryFilter.java Co-authored-by: Anonymitaet <50226895+Anonymitaet@users.noreply.github.com>

Update pulsar-broker/src/main/java/org/apache/pulsar/broker/service/E…

1c06570

…ntryFilter.java Co-authored-by: Anonymitaet <50226895+Anonymitaet@users.noreply.github.com>

Address comment

7e9f5b5

Merge remote-tracking branch 'origin/plugin-filter' into plugin-filter

40a86ba

fix check style

02c1e62

eolivelli requested changes Nov 26, 2021

View reviewed changes

315157973 added 2 commits November 26, 2021 15:17

split into different class

efb7078

fix check style

313d402

315157973 added 2 commits November 26, 2021 15:52

fix unit test

0d9f1a7

fix unit test

b5611dd

codelipenghui reviewed Nov 26, 2021

View reviewed changes

315157973 added 3 commits November 26, 2021 17:33

address comments

b428669

change doc

d69a0fd

fix check style

e7c51a4

codelipenghui approved these changes Nov 26, 2021

View reviewed changes

codelipenghui added this to the 2.10.0 milestone Nov 26, 2021

codelipenghui requested a review from merlimat November 26, 2021 13:02

eolivelli requested changes Nov 26, 2021

View reviewed changes

add close

e16c16b

eolivelli requested changes Nov 30, 2021

View reviewed changes

Add unit test for closing

74aea66

eolivelli approved these changes Nov 30, 2021

View reviewed changes

eolivelli merged commit dac47cb into apache:master Nov 30, 2021

fxbing pushed a commit to fxbing/pulsar that referenced this pull request Dec 19, 2021

[PIP-105] Part-2 Support pluggable entry filter in Dispatcher (apache…

6166b44

…#12970)

Anonymitaet mentioned this pull request Jan 6, 2022

[Doc] add explanations for entry filter #13646

Merged

Anonymitaet added doc-complete Your PR changes impact docs and the related docs have been already added. and removed doc-required Your PR changes impact docs and you will update later. labels Jan 6, 2022

315157973 added the release/2.10.0 label Jan 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PIP-105] Part-2 Support pluggable entry filter in Dispatcher #12970

[PIP-105] Part-2 Support pluggable entry filter in Dispatcher #12970

315157973 commented Nov 25, 2021

codelipenghui Nov 25, 2021

codelipenghui Nov 25, 2021

Anonymitaet left a comment

eolivelli left a comment

eolivelli Nov 26, 2021

315157973 Nov 26, 2021

315157973 commented Nov 26, 2021

315157973 commented Nov 26, 2021

eolivelli left a comment

315157973 commented Nov 27, 2021

315157973 commented Nov 29, 2021

wolfstudy commented Nov 30, 2021

eolivelli left a comment

[PIP-105] Part-2 Support pluggable entry filter in Dispatcher #12970

[PIP-105] Part-2 Support pluggable entry filter in Dispatcher #12970

Conversation

315157973 commented Nov 25, 2021

Motivation

Modifications

Documentation

codelipenghui Nov 25, 2021

Choose a reason for hiding this comment

codelipenghui Nov 25, 2021

Choose a reason for hiding this comment

Anonymitaet left a comment

Choose a reason for hiding this comment

eolivelli left a comment

Choose a reason for hiding this comment

eolivelli Nov 26, 2021

Choose a reason for hiding this comment

315157973 Nov 26, 2021

Choose a reason for hiding this comment

315157973 commented Nov 26, 2021

315157973 commented Nov 26, 2021

eolivelli left a comment

Choose a reason for hiding this comment

315157973 commented Nov 27, 2021

315157973 commented Nov 29, 2021

wolfstudy commented Nov 30, 2021

eolivelli left a comment

Choose a reason for hiding this comment