feat: kafka-connect-monitor service

[epipav]

Summary

• Adds a new Kafka Connect monitoring service that exposes Prometheus metrics for connector and task health
• Provides real-time health monitoring capabilities for Kafka Connect via /connector-health endpoint
• Enables alerting and observability for Kafka Connect infrastructure
• It is running as a sidecar to kafka-connect service

Implementation Details

Created a new standalone service kafka-connect-monitor that:

• Polls Kafka Connect REST API at localhost:8083 to fetch connector statuses
• Exposes Prometheus metrics using one-hot encoding for connector and task states (RUNNING, FAILED, PAUSED, UNASSIGNED)
• Runs on port 8085 with health check endpoint at /health
• Built with Express and prom-client for metrics collection

[github-actions]

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

• feat: add user authentication (CM-123)
• feat: add user authentication (IN-123)

Projects:

• CM: Community Data Platform
• IN: Insights

Please add a Jira issue key to your PR title.

[github-actions]

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

• feat: add user authentication (CM-123)
• feat: add user authentication (IN-123)

Projects:

• CM: Community Data Platform
• IN: Insights

Please add a Jira issue key to your PR title.

[github-actions]

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

• feat: add user authentication (CM-123)
• feat: add user authentication (IN-123)

Projects:

• CM: Community Data Platform
• IN: Insights

Please add a Jira issue key to your PR title.

[cursor]

Bug: Middleware Order Causes Error Handling Failure

The errorMiddleware tries to use req.log for error reporting. However, loggingMiddleware is placed after express.json(). If JSON parsing fails, req.log won't be initialized, leading to a TypeError when the error handler attempts to access req.log.error().

Additional Locations (1)

• services/apps/kafka_connect_monitor/src/main.ts#L17-L24