<Developer's Posts>
this project is my older extract transform broker project.
I moved my git account to my current account and then transferred my previously open project etl-for-kafka to my current account.
In this project, I did not use any libraries other than the kafka consumer and the basic JDBC libraries.
Because this project was a challenge for me to make 'extract, transform, load' at the framework level without using the library.
- Set up Database information
config.properties
mysql.host =jdbc:mysql://127.0.0.1:3306
mysql.dbname =event
mysql.username =admin
mysql.password =password
- For local testing, run docker-compose up.
docker-compose up
- Consumer Setting
we can now set the Kafka consumer. However, if you want, you can also create other consumers to match the consumer interface.
- Transform Setting
we can now set the DeduplicationTransForm. It filters duplicate data based on the key value.
If you keep the Transform interface, you can add a variety of transforms.
- Window Setting
we can now set the EventSlidingWindow. It is a moving window with duration.
- Output Setting
This is the target database to load the data into. It currently supports Mysql.
new StreamContext
.StreamContextBuilder()
.setConsumer(new KafkaConsumerImpl.KafkaConsumerBuilder(topics).build())
.setTransform(new DeduplicationTransForm(filterByKeys, targetWindow))
.setWindow(new EventSlidingWindow(TEN_MINUTE))
.setOutput(new EventMySqlRepository())
.startPipeline();
