Skip to content

Extract encoding part of KafkaStreamWriter to transformer component #129

@kevinwallimann

Description

@kevinwallimann

KafkaStreamWriter should not be dependent on Abris (confluent and Avro)
Thus, the encoding part should be extracted to a preceding transformer component

A new transformer should be created, e.g. za.co.absa.hyperdrive.ingestor.implementation.transformer.avro.ConfluentAvroEncoderTransformer

Breaking changes
Configuration properties will need to be adjusted

How to migrate Hyperdrive-Trigger

  1. Replace
"component.writer

with

"component.transformer.id.2=confluent.avro.encoder", "component.transformer.class.confluent.avro.encoder=za.co.absa.hyperdrive.ingestor.implementation.transformer.avro.ConfluentAvroEncoderTransformer",
"component.writer
  1. Replace
writer.kafka.schema

with

transformer.confluent.avro.encoder.schema
  1. Replace
writer.kafka.value

with

transformer.confluent.avro.encoder.value
  1. Replace
writer.kafka.produce.keys

with

transformer.confluent.avro.encoder.produce.keys
  1. Replace
writer.kafka.key

with

transformer.confluent.avro.encoder.key
  1. Replace
writer.kafka.option

with

transformer.confluent.avro.encoder.option

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions