debezium

• https://www.baeldung.com/debezium-intro
• https://www.mastertheboss.com/jboss-frameworks/debezium/getting-started-with-debezium/

Examples

• https://github.com/rbiedrawa/cdc-postgres
• https://github.com/debezium/debezium-examples/blob/main/tutorial/docker-compose-postgres.yaml
• https://medium.com/@alexander.murylev/kafka-connect-debezium-mysql-source-sink-replication-pipeline-fb4d7e9df790

Architecture

Implementation Debezium

• Kafka Connect
• Debezium Embedded Engine and
• Debezium Server

Kafka Connect is a framework and runtime for implementing and operating:

• Source connectors such as Debezium that send records into Kafka
• Sink connectors that propagate records from Kafka topics to other systems
  • https://github.com/debezium/debezium-examples/blob/main/tutorial/debezium-with-oracle-jdbc/Dockerfile
  • https://www.mastertheboss.com/jboss-frameworks/debezium/tutorial-using-debezium-jdbc-connector/

Kafka Connect operates as a separate service besides the Kafka broker.

Debezium Server

• https://github.com/debezium/debezium-examples/tree/main/kinesis
• https://github.com/debezium/debezium-examples/tree/main/debezium-server/debezium-server-sink-pubsub
• https://infinitelambda.com/postgres-cdc-debezium-google-pubsub/
• https://stackoverflow.com/questions/72614792/how-to-apply-debezium-cdc-events-from-pub-sub-onto-a-database

The Debezium server is a configurable, ready-to-use application that streams change events from a source database to a variety of messaging infrastructures.

Debezium Engine

library embedded into your custom Java applications.

Example

Debezium Kafka Connect configuration for PostgreSQL

{
"name": "visits-connector",
  "config": {
    "connector.class":
    "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres",
    "database.port": "5432",
    "database.user": "postgres",
    "database.password": "postgres",
    "database.dbname" : "postgres",
    "database.server.name": "dbserver1",
    "schema.include.list": "dedp_schema",
    "topic.prefix": "dedp"
  }
}

It defines the connection parameters, all the schemas to include in the watching operation, and finally the prefix for the created topic for each synchronized table. As a result, if there is a dedp_schema.events table, the connector will write all the changes to the dedp.dedp_schema.events topic.