Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support key_based batch builder for functions and sources #8523

Merged
merged 7 commits into from
Nov 16, 2020
Merged

Support key_based batch builder for functions and sources #8523

merged 7 commits into from
Nov 16, 2020

Conversation

wolfstudy
Copy link
Member

Signed-off-by: xiaolong.ran rxl@apache.org

Motivation

Currently, we support the Key_Shared subscription mode in Pulsar Function. In order to ensure that when batch is turned on, we can also ensure that messages are distributed to different consumers in the correct order, so we need to support the batch builder of KEY_BASED in Pulsar Functions.

Modifications

  • Add --batch-builder for Pulsar Functions
  • Add --batch-builder for Pulsar Sources
  • Add test case

@wolfstudy
Support key_based batch builder for functions and sources
efa2473 
Signed-off-by: xiaolong.ran <rxl@apache.org>
@wolfstudy wolfstudy self-assigned this Nov 11, 2020
@wolfstudy wolfstudy added this to the 2.7.0 milestone Nov 11, 2020
@wolfstudy
Avoid using * in import
9b93663 
Signed-off-by: xiaolong.ran <rxl@apache.org>
@wolfstudy wolfstudy mentioned this pull request Nov 11, 2020
Closed
@wolfstudy
Copy link
Member Author

/pulsarbot run-failure-checks

@wolfstudy
fix ci error
ac089ea 
Signed-off-by: xiaolong.ran <rxl@apache.org>
@wolfstudy wolfstudy mentioned this pull request Nov 12, 2020
Merged
sijie
sijie requested changes Nov 12, 2020
View reviewed changes
Copy link
Member

@sijie sijie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good. However, I didn't see any test case. Can you add one?

@wolfstudy
fix ci error
8bd51a3 
Signed-off-by: xiaolong.ran <rxl@apache.org>
@wolfstudy
Copy link
Member Author

Overall looks good. However, I didn't see any test case. Can you add one?

The current unit tests can be covered. The change of this pull request is to add a batchBuilder field to ProducerConfig. The current unit test framework covers the Pojo object ProducerConfig, so we only need to add this field to the existing unit test framework. Refer to pulsar-functions/utils/src/test/java/org/apache/pulsar/functions/utils/SourceConfigUtilsTest.java and pulsar-functions/utils/src/test/java/org/apache/pulsar/functions/utils/FunctionConfigUtilsTest.java.

@wolfstudy
Copy link
Member Author

You mean, do we add another integration test on top of the unit test?

@wolfstudy
Add test case
8578e3a 
Signed-off-by: xiaolong.ran <rxl@apache.org>
@wolfstudy
Add test case
70cdced 
Signed-off-by: xiaolong.ran <rxl@apache.org>
@wolfstudy
Copy link
Member Author

@sijie PTAL again thanks. I added the --batch-builder parameter to the existing integration test to confirm whether this parameter is correctly identified.

@sijie sijie merged commit 8da9422 into apache:master Nov 16, 2020
flowchartsman pushed a commit to flowchartsman/pulsar that referenced this pull request Nov 17, 2020
@wolfstudy @flowchartsman
Support key_based batch builder for functions and sources (apache#8523)
0d3bf5e 
### Motivation

Currently, we support the Key_Shared subscription mode in Pulsar Function. In order to ensure that when batch is turned on, we can also ensure that messages are distributed to different consumers in the correct order, so we need to support the batch builder of `KEY_BASED` in Pulsar Functions.

### Modifications

- Add `--batch-builder` for Pulsar Functions
- Add `--batch-builder` for Pulsar Sources
- Add test case
codelipenghui pushed a commit that referenced this pull request Nov 22, 2020
@wolfstudy
Support key base for python function (#8540)
334dffd 
### Motivation

The changing base on #8523 

### Modifications

Support key_base  for python function
@wolfstudy wolfstudy mentioned this pull request Dec 1, 2020
Merged
wolfstudy added a commit that referenced this pull request Dec 3, 2020
@wolfstudy
Support key based batch builder for Go Functions (#8761)
c3b785a 
Signed-off-by: xiaolong.ran <rxl@apache.org>

### Motivation

The changing base on #8523

### Modifications

Support key_based batch builder for Go Functions
RobertIndie pushed a commit to RobertIndie/pulsar that referenced this pull request Dec 8, 2020
@wolfstudy @RobertIndie
Support key based batch builder for Go Functions (apache#8761)
c2ab9e4 
Signed-off-by: xiaolong.ran <rxl@apache.org>

### Motivation

The changing base on apache#8523

### Modifications

Support key_based batch builder for Go Functions
@lhotari lhotari mentioned this pull request Aug 18, 2021
Merged
@lhotari lhotari mentioned this pull request Aug 18, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sijie sijie sijie approved these changes

@srkukarni srkukarni Awaiting requested review from srkukarni

@jerrypeng jerrypeng Awaiting requested review from jerrypeng

@jiazhai jiazhai Awaiting requested review from jiazhai

@codelipenghui codelipenghui Awaiting requested review from codelipenghui

Assignees

@wolfstudy wolfstudy

Labels
area/function
Projects
None yet
Milestone
2.7.0
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@wolfstudy @sijie