Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pulsar supporting DLQ for sources/sinks #7032

Closed
ADHB opened this issue May 25, 2020 · 3 comments · Fixed by #7094 or #7339
Closed

Pulsar supporting DLQ for sources/sinks #7032

ADHB opened this issue May 25, 2020 · 3 comments · Fixed by #7094 or #7339
Assignees
@315157973
Labels
area/connector type/feature The PR added a new feature or issue requested a new feature
Milestone
2.6.0

Comments

@ADHB
Copy link

ADHB commented May 25, 2020 •
edited
Loading

Currently Pulsar doesn't support DLQ for sources and sinks. Only functions support DLQ.
It is very important for sinks like ElasticSearch which is widely used in production env for logging systems to be able to redirect unacknowledged messages to another topic , keeping otherwise the brokers running as normal.

I would like to see a setting in consumer.conf or broker.conf that enables DLQ on a Pulsar configuration, so DLQ will be turned on for sources, sinks or any functions.

or

I would like to create a sink with an extra flag --DLQ parameter. Giving a topic to the parameter will turn on DLQ.
@ADHB ADHB added the type/feature The PR added a new feature or issue requested a new feature label May 25, 2020
@sijie sijie mentioned this issue May 25, 2020
Closed
@codelipenghui
Copy link
Contributor

codelipenghui commented May 25, 2020 •
edited
Loading

Thanks for your feedback, @315157973 is working on this issue.

@codelipenghui codelipenghui added this to the 2.6.0 milestone May 25, 2020
@315157973
Copy link
Contributor

315157973 commented May 27, 2020 •
edited
Loading

Please assign it to me @codelipenghui

@315157973 315157973 mentioned this issue May 29, 2020
Merged
1 task
jiazhai pushed a commit that referenced this issue May 30, 2020
@315157973
fix Pulsar supporting DLQ for sources/sinks #7032 (#7094)
36df3bb 
Fixes #7032

### Motivation
Source data flow:
Custom Source-> Function-> Pulsar producer-> Broker

Sink data flow:
Broker-> Pulsar consumer-> Function-> Custom Sink

Pulsar consumer is only used in sink mode, so it is necessary to add DLQ parameters for sink mode
@ADHB
Copy link
Author

ADHB commented May 30, 2020 via email

Huanli-Meng pushed a commit to Huanli-Meng/pulsar that referenced this issue Jun 1, 2020
@315157973 @Huanli-Meng
fix Pulsar supporting DLQ for sources/sinks apache#7032 (apache#7094)
5effd38 
Fixes apache#7032

### Motivation
Source data flow:
Custom Source-> Function-> Pulsar producer-> Broker

Sink data flow:
Broker-> Pulsar consumer-> Function-> Custom Sink

Pulsar consumer is only used in sink mode, so it is necessary to add DLQ parameters for sink mode
Huanli-Meng pushed a commit to Huanli-Meng/pulsar that referenced this issue Jun 1, 2020
@315157973 @Huanli-Meng
fix Pulsar supporting DLQ for sources/sinks apache#7032 (apache#7094)
458cb80 
Fixes apache#7032

### Motivation
Source data flow:
Custom Source-> Function-> Pulsar producer-> Broker

Sink data flow:
Broker-> Pulsar consumer-> Function-> Custom Sink

Pulsar consumer is only used in sink mode, so it is necessary to add DLQ parameters for sink mode
@sijie sijie mentioned this issue Jun 10, 2020
Closed
Huanli-Meng pushed a commit to Huanli-Meng/pulsar that referenced this issue Jun 12, 2020
@315157973 @Huanli-Meng
fix Pulsar supporting DLQ for sources/sinks apache#7032 (apache#7094)
95137ea 
Fixes apache#7032

### Motivation
Source data flow:
Custom Source-> Function-> Pulsar producer-> Broker

Sink data flow:
Broker-> Pulsar consumer-> Function-> Custom Sink

Pulsar consumer is only used in sink mode, so it is necessary to add DLQ parameters for sink mode
@Huanli-Meng Huanli-Meng mentioned this issue Jun 23, 2020
Merged
huangdx0726 pushed a commit to huangdx0726/pulsar that referenced this issue Aug 24, 2020
@315157973 @huangdx0726
fix Pulsar supporting DLQ for sources/sinks apache#7032 (apache#7094)
a5b8c7b 
Fixes apache#7032

### Motivation
Source data flow:
Custom Source-> Function-> Pulsar producer-> Broker

Sink data flow:
Broker-> Pulsar consumer-> Function-> Custom Sink

Pulsar consumer is only used in sink mode, so it is necessary to add DLQ parameters for sink mode
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@315157973 315157973

Labels
area/connector type/feature The PR added a new feature or issue requested a new feature
Projects
None yet
Milestone
2.6.0
5 participants
@sijie @315157973 @codelipenghui @tisonkun @ADHB