Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

function to read compacted topics #7193

Merged
merged 6 commits into from
Jul 2, 2020
Merged

Conversation

315157973
Copy link
Contributor

Fixes #5538

Motivation

Modifications

In function mode and sink mode, PulsarSource can read compacted topic。
By inputSpecs parameter, each topic can independently decide whether to read compacted

Verifying this change

unit test:
org.apache.pulsar.io.PulsarFunctionE2ETest#testReadCompactedFunction
org.apache.pulsar.io.PulsarFunctionE2ETest#testReadCompactedSink

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API: (yes / no)
  • The schema: (yes / no / don't know)
  • The default values of configurations: (yes / no)
  • The wire protocol: (yes / no)
  • The rest endpoints: (yes / no)
  • The admin cli options: (yes / no)
  • Anything that affects deployment: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
  • If a feature is not applicable for documentation, explain why?
  • If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

@jiazhai jiazhai requested a review from wolfstudy June 24, 2020 01:22
@jiazhai
Copy link
Member

jiazhai commented Jun 24, 2020

@jerrypeng @sijie Would you please help review this PR again?

@jiazhai
Copy link
Member

jiazhai commented Jun 24, 2020

/pulsarbot run-failure-checks

@codelipenghui codelipenghui merged commit 7a80ca9 into apache:master Jul 2, 2020
@315157973 315157973 deleted the compacted branch July 2, 2020 10:51
huangdx0726 pushed a commit to huangdx0726/pulsar that referenced this pull request Aug 24, 2020
Fixes apache#5538

### Motivation



### Modifications
In function mode and sink mode, PulsarSource can read compacted topic。
By `inputSpecs` parameter, each topic can independently decide whether to read compacted

### Verifying this change
unit test:
org.apache.pulsar.io.PulsarFunctionE2ETest#testReadCompactedFunction
org.apache.pulsar.io.PulsarFunctionE2ETest#testReadCompactedSink
@jdimeo
Copy link

jdimeo commented May 3, 2022

How does one configure their function to read from a compacted topic? the current documentation doesn't seem to mention it and it's not obvious looking at the code of this PR. Does it happen automatically? does readCompacted: true go into the user config properties?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Pulsar function to read compacted topics
6 participants