The Ml User Delete service is constructed upon the framework of Flink, Kafka and Mongo, a powerful approach for updating and managing data in a scalable and real-time manner.
- Apache Kafka is a distributed event streaming platform that is designed for handling high volumes of real-time data. In this service, Kafka generates user data (userId, organisationId) as a JSON event when a user chooses to remove their account from the Sunbird platform.
- Apache Flink is a stream processing framework that provides high-throughput, low-latency, and exactly-once processing of streaming data. Flink uses the Kafka event and establishes a connection with the Mongo database to update documents from various collections.
- Mongo Database is a distributed NoSQL database that is designed to handle massive amounts of data across many commodity servers, ensuring high availability and fault tolerance. User data containing user personally identifiable information will be updated.
Configuration variables:
| Variable | Default Value | Purpose |
|---|---|---|
| kafka.input.topic | {{env}}.delete.user | Kafka topic from which messages/events are read to be processed. |
| kafka.groupId | {{env}}ml-user-delete-group | Kafka input topic group Id |