Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pulsar IO - KafkaSource - allow to manage Avro Encoded messages #4

Closed
wants to merge 24 commits into from
Closed

Pulsar IO - KafkaSource - allow to manage Avro Encoded messages #4

wants to merge 24 commits into from

Conversation

eolivelli
Copy link
Owner

@eolivelli eolivelli commented Feb 3, 2021 •
edited
Loading

Summary of changes:

  • add new AvroRecordKafkaSource that produces GenericRecords to the Pulsar topic
  • support Schema Evolution end-to-end (i.e. add fields to the original schema in the Kafka world, and see the new fields in the Pulsar topic, without any reconfiguration or restart)
  • add Confluent Schema Registry Client to the Kafka Connector NAR
  • the configuration of the Schema Registry Client is to be done in the consumerProperties property of the source
  • allow PulsarSink to deal with org.apache.pulsar.client.api.schema.GenericRecord (it must not enforce an empty schema to the topic while starting the source)
  • allow AvroWriter to deal with GenericRecord that are not a subclass of AvroGenericRecord
  • add integration tests with Kafka and Schema Registry

This patch includes this patch that is to be committed as pre-requisite
apache#9396

@eolivelli
Implement Avro support for Kafka Source
28cb99c
@eolivelli
Merge branch 'master' into impl/kafka-schema-aware
1595182
@eolivelli
make it work
8051cd5
@eolivelli
clean up
ae87b9f
@eolivelli
more cleanup
68d1346
@eolivelli
cache schema
9b76f47
@eolivelli
fix compression bug
490b631
@eolivelli
Compression must be applied during deferred schema preparation and en…
047cadf 
…ableBatching is enabled

if you do not set an initial schema to the Producer the schema must be prepared at the first message with a Schema.
There is a bug and compression is not applied in this case, and the consumer receives an uncompressed message, failing
@eolivelli
Merge brach fix/prod-deferred-schema-compressed
82cb6e7
@eolivelli
Merge branch 'master' into impl/kafka-schema-aware
377ca2c
@eolivelli
clean up
f77c00b
@eolivelli
Merge branch 'fix/prod-deferred-schema-compressed' into impl/kafka-sc…
fbefdfa 
…hema-aware
@eolivelli
fix config
7f4fe01
@eolivelli
Merge remote-tracking branch 'origin/master' into impl/kafka-schema-a…
3a79356 
…ware
@eolivelli
add tests
b5f7256
@eolivelli
add test
dbc6de7
@eolivelli
Merge remote-tracking branch 'origin/master' into impl/kafka-schema-a…
b6a6525 
…ware-with-tests
@eolivelli
make schemaregistry start
b42475a
@eolivelli
Merge remote-tracking branch 'origin/master' into impl/kafka-schema-a…
48efa13 
…ware-with-tests
@eolivelli
use kafka cli tools
d36f73f
@eolivelli
test works
311b16b
@eolivelli
Merge remote-tracking branch 'origin/master' into impl/kafka-schema-a…
368bd44 
…ware-with-tests
@eolivelli
clean up
2968afb
@eolivelli
more cleanup
0a6c456
eolivelli
eolivelli commented Feb 3, 2021
View reviewed changes
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
* @param payload
* @return a new payload
*/
private ByteBuf applyCompression(ByteBuf payload) {
Copy link
Owner Author

@eolivelli eolivelli Feb 3, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all of the changes in ProducerImpl are part of apache#9396
that is required for this patch to work

@eolivelli eolivelli changed the title Impl/kafka schema aware with tests Pulsar IO - KafkaSource - allow to manage Avro Encoded messages Feb 3, 2021
@eolivelli eolivelli closed this Feb 3, 2021
eolivelli pushed a commit that referenced this pull request Nov 8, 2021
@dlg99
Postgres WAL does not truncate / confirmed_flush_lsn dows not change.…
21a93e9 
… Downgrading debezium to the same version as apache. (#4)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@eolivelli