Monitoring and auto creation of deleted/failed kafka connectors

[shashwat-egov]

Recently while working on national-dashboard service, we observed that kafka connectors were getting deleted. One possible suspect was the event where kafka-cluster was getting restarted. This required manual intervention from our side as we again had to create all the kafka connectors. I was thinking about a way to automate this whole process by making use of kafka connect's REST APIs.

Kafka connector monitoring service/script (this service will hold the names and configurations of all kafka connectors that are required by DIGIT services) -> GET /connectors request to kafka-connect -> The service will then check for the connectors which have failed/got deleted -> Trigger creation of absent/failed connectors.

Please list out any other approaches.

[eGovMan]

Which Kafka connectors?

Are your referring to these APIs - https://docs.confluent.io/platform/current/connect/monitoring.html?

Isn’t there a way to monitor this in Prometheus?

[shashwat-egov]

Yes, so kafka connect exposes APIs for checking the status of connectors. In case any connector goes down, we can restart it. However in cases where the connector gets deleted altogether(this was an issue that we faced multiple times during national dashboard development and testing), some other service should always keep track of the connectors which are being used by DIGIT's services and auto create them. Now, the first scenario where a connector fails can be addressed via some alert that Prometheus will generate, I am not sure if any similar alert will be generated if a connector gets deleted altogether.