SQL Server Kafka MongoDB Sync

Abstract

I created this solution to use the best of both worlds, the storage power of SQL Server and the read speed of MongoDB.

Architecture

It was built in Docker with 10 application layers divided into 5 categories:

I. Transactional Persistence Database:

1. SQL Server

II. Fast Read Database:

2. MongoDB

III. Synchronization:

3. Zookeeper
4. Kafka
5. Kafka Connect
6. Kafka UI

IV: Synchronization Configuration:

7. Replicator

V: Restful Services:

8. Restful API to SQL Server's CRUD (NestJs)
9. Restful API to Mongodb's reader (NestJs)
10. Restful API to Mongodb's reader (.NET 8)

Clone & Run

> git clone https://github.com/avm-sistemas/sqlserver-kafka-mongodb-sync.git
> cd sqlserver-kafka-mongodb-sync
> docker-compose up --build -d --force-recreate

Results:

Docker:

SQL Server:

Kafka:

Mongodb:

API Restful SQL Server

API Restful MongoDB (NestJs)

API Restful MongoDB (.NET 8)