Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kafka connect sink adaptor to support non-primitive schemas #10410

Merged
merged 4 commits into from
Jun 10, 2021
Merged

Kafka connect sink adaptor to support non-primitive schemas #10410

merged 4 commits into from
Jun 10, 2021

Conversation

dlg99
Copy link
Contributor

@dlg99 dlg99 commented Apr 27, 2021 •
edited
Loading

Motivation

Add support of non-primitive schemas (avro, json) to kafka adaptor sink.

Modifications

Added schema support.

Verifying this change

  • Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

Added unit tests

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): kind of. Confluent dependency version was listed twice in pom file, with different version. Fixed that, left the newer version.
  • The public API: no
  • The schema: no
  • The default values of configurations: no
  • The wire protocol: no
  • The rest endpoints: no
  • The admin cli options: no
  • Anything that affects deployment: no

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
  • If a feature is not applicable for documentation, explain why?
  • If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

@dlg99
Copy link
Contributor Author

dlg99 commented Apr 27, 2021

only the last commit is relevant here; the rest are from parent PR

@dlg99 dlg99 force-pushed the kafka-connect-adaptor-schema branch 2 times, most recently from dca5452 to a7db773 Compare June 2, 2021 21:23
eolivelli
eolivelli requested changes Jun 3, 2021
View reviewed changes
Copy link
Contributor

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM

I left one comment (possibly one bug, we should add a test case for KV if it is missing)
and one code style suggestion (not a big deal)

...aptor/src/main/java/org/apache/pulsar/io/kafka/connect/schema/PulsarSchemaToKafkaSchema.java Outdated Show resolved Hide resolved
...aptor/src/main/java/org/apache/pulsar/io/kafka/connect/schema/PulsarSchemaToKafkaSchema.java Outdated Show resolved Hide resolved
@dlg99 dlg99 requested a review from eolivelli June 3, 2021 15:26
eolivelli
eolivelli approved these changes Jun 3, 2021
View reviewed changes
Copy link
Contributor

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sijie the patch is ready for review

@dlg99
Copy link
Contributor Author

dlg99 commented Jun 7, 2021

@codelipenghui Can you take a look please?

@dlg99 dlg99 force-pushed the kafka-connect-adaptor-schema branch from eae35f2 to 7593950 Compare June 8, 2021 16:20
@eolivelli eolivelli added this to the 2.9.0 milestone Jun 9, 2021
@eolivelli eolivelli merged commit cf50da3 into apache:master Jun 10, 2021
@merlimat merlimat modified the milestones: 2.9.0, 2.8.0 Jun 10, 2021
codelipenghui pushed a commit that referenced this pull request Jun 11, 2021
@dlg99 @codelipenghui
Kafka connect sink adaptor to support non-primitive schemas (#10410)
3505b98 
(cherry picked from commit cf50da3)
eolivelli pushed a commit to datastax/pulsar that referenced this pull request Jun 14, 2021
@dlg99 @eolivelli
Kafka connect sink adaptor to support non-primitive schemas (apache#1…
4eb2321 
…0410)

(cherry picked from commit cf50da3)
(cherry picked from commit 3505b98)
yangl pushed a commit to yangl/pulsar that referenced this pull request Jun 23, 2021
@dlg99 @yangl
Kafka connect sink adaptor to support non-primitive schemas (apache#1…
42357bd 
…0410)
@dlg99 dlg99 deleted the kafka-connect-adaptor-schema branch October 14, 2021 23:30
bharanic-dev pushed a commit to bharanic-dev/pulsar that referenced this pull request Mar 18, 2022
@dlg99 @ciaocloud
Kafka connect sink adaptor to support non-primitive schemas (apache#1…
1bf1dde 
…0410)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@eolivelli eolivelli eolivelli approved these changes

@codelipenghui codelipenghui codelipenghui approved these changes

@sijie sijie Awaiting requested review from sijie

@congbobo184 congbobo184 Awaiting requested review from congbobo184

@jerrypeng jerrypeng Awaiting requested review from jerrypeng

@merlimat merlimat Awaiting requested review from merlimat

Assignees

@dlg99 dlg99

Labels
None yet
Projects
None yet
Milestone
2.8.0
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@dlg99 @eolivelli @codelipenghui @merlimat