Sink<GenericObject> unwrap internal AutoConsumeSchema and allow to handle topics with KeyValue schema #10211
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ting it at build time (Sink<GenericObject>)
|
/pulsarbot rerun-failure-checks |
sijie
approved these changes
Apr 13, 2021
|
/pulsarbot rerun-failure-checks |
codelipenghui
approved these changes
Apr 15, 2021
|
I am adding debug in order to see what's wrong in CI. |
|
@codelipenghui I finally figured out what was wrong in the integration test. now it passes locally. |
eolivelli
added a commit
to datastax/pulsar
that referenced
this pull request
May 12, 2021
…ndle topics with KeyValue schema (apache#10211)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
When you run a Sink (or Sink) you are using AutoConsumeSchema and AutoConsumeSchema is a wrapper around the actual schema attached to the topic.
In case of a topic with KeyValueSchema you are not able to access the Key Schema and the Value Schema because Record#getSchema returns the AutoConsumeSchema and not the KeyValueSchema itself.
Modifications
The test sets a property on the message with the expected schema type, this way the sink breaks in case of some error in the system and bad detection of the schema.
Verifying this change
This change added tests
Documentation
I am going to write documentation about how to write Pulsar IO Sinks that work with any schema without binding to a specific schema at compile time (Sink)