Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Schema Registry support through PulsarKafkaProducer #4228

Closed
briandoran opened this issue May 7, 2019 · 0 comments · Fixed by #4534
Closed

Schema Registry support through PulsarKafkaProducer #4228

briandoran opened this issue May 7, 2019 · 0 comments · Fixed by #4534
Labels
type/feature The PR added a new feature or issue requested a new feature

Comments

@briandoran
Copy link

Is your feature request related to a problem? Please describe.
In order to minimise integration issues with Kafka/Pulsar I am wondering if its possible to use the Apache Kafka producer API to specify using AVRO schemas in Pulsar? It looks like this can just be done through the pulsar ProducerBuilder API pulsarClient.newProducer(avro).create(); which returns a pulsar.Producer rather than a apache.kafka.Producer. I wanted to keep changes to integration to a minimum which is fine when not specifying a schema. new PulsarKafkaProducer<>(props) vs new KafkaProducer<>(props)

Describe the solution you'd like
As discussed with @sijie on Slack a perfect solution would be adding a new Schema parameter to the PulsarKafkaProducer?

new PulsarKafkaProducer<>(props, schema)

This would allow the schema will be passed down to the internal pulsar client.

We also spoke about a integration with external schema registry, and implementing a ConfluentSchema which talks to confluent schema registry. so it will use confluent schema registry and confluent converters to do serialization and deserialization.

so the code would be something like:

`Schema schema = ConfluentSchema.builder()
.withSchemaRegistryUrl(...)
.withKeySerializer(...)
.withValueSerializer(...);

Producer<> producer = new PulsarKafkaProducer(props, schema);`

Great work on the support BTW, the slack portal is invaluable for getting great information with quick responses.

@briandoran briandoran added the type/feature The PR added a new feature or issue requested a new feature label May 7, 2019
sijie pushed a commit that referenced this issue Jul 7, 2019
Fixes #4228

Master Issue: #4228

### Motivation

Use Pulsar schema in pulsar kafka client.

### Modifications

Support schema of pulsar for pulsar kafka client

### Verifying this change

Add Unit test
easyfan pushed a commit to easyfan/pulsar that referenced this issue Jul 26, 2019
Fixes apache#4228

Master Issue: apache#4228

### Motivation

Use Pulsar schema in pulsar kafka client.

### Modifications

Support schema of pulsar for pulsar kafka client

### Verifying this change

Add Unit test
jiazhai pushed a commit that referenced this issue Aug 28, 2019
Fixes #4228

Master Issue: #4228

### Motivation

Use Pulsar schema in pulsar kafka client.

### Modifications

Support schema of pulsar for pulsar kafka client

### Verifying this change

Add Unit test
(cherry picked from commit 99d3c11)
sijie pushed a commit to apache/pulsar-adapters that referenced this issue Nov 6, 2020
Fixes apache/pulsar#4228

Master Issue: apache/pulsar#4228

### Motivation

Use Pulsar schema in pulsar kafka client.

### Modifications

Support schema of pulsar for pulsar kafka client

### Verifying this change

Add Unit test
sijie pushed a commit to apache/pulsar-adapters that referenced this issue Nov 6, 2020
Fixes apache/pulsar#4228

Master Issue: apache/pulsar#4228

### Motivation

Use Pulsar schema in pulsar kafka client.

### Modifications

Support schema of pulsar for pulsar kafka client

### Verifying this change

Add Unit test
sijie pushed a commit to apache/pulsar-release that referenced this issue Nov 12, 2020
Fixes apache/pulsar#4228

Master Issue: apache/pulsar#4228

### Motivation

Use Pulsar schema in pulsar kafka client.

### Modifications

Support schema of pulsar for pulsar kafka client

### Verifying this change

Add Unit test
sijie pushed a commit to apache/pulsar-release that referenced this issue Nov 12, 2020
Fixes apache/pulsar#4228

Master Issue: apache/pulsar#4228

### Motivation

Use Pulsar schema in pulsar kafka client.

### Modifications

Support schema of pulsar for pulsar kafka client

### Verifying this change

Add Unit test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/feature The PR added a new feature or issue requested a new feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants