A Kafka Connect SMT that allows you to add the record key to the value as a named field.
The KeyToField transformation is designed to enhance Kafka Connect functionality by including the record key as a field within the record's value. This can be particularly useful in scenarios where downstream systems require access to the original key alongside the record data.
- Add the record key to the value as a named field.
- Customizable field name and delimiter.
- Use the latest release on GitHub, or build the JAR file from source using Maven:
mvn clean package- Copy the generated JAR file (
keytofield-transform-<version>.jar) to the Kafka Connect plugins directory. - Restart Kafka Connect for the reload the plugin directory.
- Update your connector with the SMT configuration
The KeyToField transformation can be configured with the following properties:
field.name: Name of the field to insert the Kafka key to (default:kafkaKey).field.delimiter: Delimiter to use when concatenating the key fields (default:-).
To use the KeyToField transformation, add it to your Kafka Connect connector configuration:
transforms=keyToField
transforms.keyToField.type=com.github.eladleev.kafka.connect.transform.keytofield.KeyToFieldTransform
transforms.keyToField.field.name=primaryKey
transforms.keyToField.field.delimiter=_
Consider a Kafka topic with the following record:
{
"key": {
"id": 123,
"timestamp": 1644439200000
},
"value": {
"data": "example"
}
}After applying the KeyToField transformation, the record will be transformed as follows:
{
"key": {
"id": 123,
"timestamp": 1644439200000
},
"value": {
"data": "example",
"primaryKey": "123_1644439200000"
}
}For your convenience, under dev/ you can find a docker-compose file that contains all necessary components for local development and testing. Kafka Connect will automatically load the connector from the target/ directory.
Use the attached bash script to submit a new Kafka Connect connector. Adminer can be used to ingest new data to the database, reflected by an event in Kafka.
Please read CONTRIBUTING.md for details of submitting a pull requests.
This project is licensed under the Apache License - see the LICENSE file for details.