Pulsar function to read compacted topics #5538
Assignees
Labels
Comments
candlerb
added
the
type/feature
|
Please assign it to me @codelipenghui |
codelipenghui
pushed a commit
that referenced
this issue
Jul 2, 2020
Fixes #5538 ### Motivation ### Modifications In function mode and sink mode, PulsarSource can read compacted topic。 By `inputSpecs` parameter, each topic can independently decide whether to read compacted ### Verifying this change unit test: org.apache.pulsar.io.PulsarFunctionE2ETest#testReadCompactedFunction org.apache.pulsar.io.PulsarFunctionE2ETest#testReadCompactedSink
huangdx0726
pushed a commit
to huangdx0726/pulsar
that referenced
this issue
Aug 24, 2020
Fixes apache#5538 ### Motivation ### Modifications In function mode and sink mode, PulsarSource can read compacted topic。 By `inputSpecs` parameter, each topic can independently decide whether to read compacted ### Verifying this change unit test: org.apache.pulsar.io.PulsarFunctionE2ETest#testReadCompactedFunction org.apache.pulsar.io.PulsarFunctionE2ETest#testReadCompactedSink
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
Pulsar functions cannot currently be configured to read the compacted version of a topic.
Describe the solution you'd like
An option to
pulsar-admin functions create/update, and/or setting in--function-config-file, to request subscribing to the compacted version of a topic.Note: a function can have multiple topics as inputs. There are already other options which are on a per-input basis (e.g.
custom-schema-inputs,custom-serde-inputs) so it ought to work like those.Describe alternatives you've considered
It occurred to me that subscription options could be part of the topic URI, e.g.
... but that would be a major change if deployed everywhere that subscriptions are used.
Additional context
A similar option might be required for sinks.
PR #5532 adds a similar feature to start a sink from "earliest" rather than "latest" message. If that's just for sinks then it would be useful for functions too.
Hence there seems to be a more general case use around setting initial subscription options for both functions and sinks.
The text was updated successfully, but these errors were encountered: