-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Schema Registry support through PulsarKafkaProducer #4228
Labels
type/feature
The PR added a new feature or issue requested a new feature
Comments
briandoran
added
the
type/feature
The PR added a new feature or issue requested a new feature
label
May 7, 2019
sijie
pushed a commit
that referenced
this issue
Jul 7, 2019
easyfan
pushed a commit
to easyfan/pulsar
that referenced
this issue
Jul 26, 2019
Fixes apache#4228 Master Issue: apache#4228 ### Motivation Use Pulsar schema in pulsar kafka client. ### Modifications Support schema of pulsar for pulsar kafka client ### Verifying this change Add Unit test
sijie
pushed a commit
to apache/pulsar-adapters
that referenced
this issue
Nov 6, 2020
Fixes apache/pulsar#4228 Master Issue: apache/pulsar#4228 ### Motivation Use Pulsar schema in pulsar kafka client. ### Modifications Support schema of pulsar for pulsar kafka client ### Verifying this change Add Unit test
sijie
pushed a commit
to apache/pulsar-adapters
that referenced
this issue
Nov 6, 2020
Fixes apache/pulsar#4228 Master Issue: apache/pulsar#4228 ### Motivation Use Pulsar schema in pulsar kafka client. ### Modifications Support schema of pulsar for pulsar kafka client ### Verifying this change Add Unit test
sijie
pushed a commit
to apache/pulsar-release
that referenced
this issue
Nov 12, 2020
Fixes apache/pulsar#4228 Master Issue: apache/pulsar#4228 ### Motivation Use Pulsar schema in pulsar kafka client. ### Modifications Support schema of pulsar for pulsar kafka client ### Verifying this change Add Unit test
sijie
pushed a commit
to apache/pulsar-release
that referenced
this issue
Nov 12, 2020
Fixes apache/pulsar#4228 Master Issue: apache/pulsar#4228 ### Motivation Use Pulsar schema in pulsar kafka client. ### Modifications Support schema of pulsar for pulsar kafka client ### Verifying this change Add Unit test
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
In order to minimise integration issues with Kafka/Pulsar I am wondering if its possible to use the Apache Kafka producer API to specify using AVRO schemas in Pulsar? It looks like this can just be done through the pulsar ProducerBuilder API
pulsarClient.newProducer(avro).create();
which returns a pulsar.Producer rather than a apache.kafka.Producer. I wanted to keep changes to integration to a minimum which is fine when not specifying a schema.new PulsarKafkaProducer<>(props)
vsnew KafkaProducer<>(props)
Describe the solution you'd like
As discussed with @sijie on Slack a perfect solution would be adding a new Schema parameter to the PulsarKafkaProducer?
new PulsarKafkaProducer<>(props, schema)
This would allow the schema will be passed down to the internal pulsar client.
We also spoke about a integration with external schema registry, and implementing a
ConfluentSchema
which talks to confluent schema registry. so it will use confluent schema registry and confluent converters to do serialization and deserialization.so the code would be something like:
`Schema schema = ConfluentSchema.builder()
.withSchemaRegistryUrl(...)
.withKeySerializer(...)
.withValueSerializer(...);
Producer<> producer = new PulsarKafkaProducer(props, schema);`
Great work on the support BTW, the slack portal is invaluable for getting great information with quick responses.
The text was updated successfully, but these errors were encountered: