fix(git-integration): improve activity emission to kafka [CM-726]

[mbani01]

This pull request improves the reliability and resource management of the Kafka queue integration in the queue_service.py and updates the repository worker to ensure proper shutdown. The main enhancements include adding robust connection retry logic, implementing a clean shutdown procedure, improving batch message sending, and making the disconnect logic safer.

Reliability and Connection Handling Improvements

• Added automatic retry logic for Kafka connection attempts using the tenacity library, with exponential backoff and limited retries, to make the service more resilient to transient connection failures. (queue_service.py) [1] [2]
• Simplified the health check for Kafka connections to reduce unnecessary metadata fetches, now checking internal producer state instead. (queue_service.py)

Resource Management and Shutdown

• Introduced a shutdown method in QueueService that flushes pending messages, ensures the producer is stopped, and cleans up resources safely during application shutdown. The repository worker now calls this method to ensure a clean shutdown. (queue_service.py, repository_worker.py) [1] [2]
• Improved the disconnect method to safely handle repeated disconnect calls by checking if the producer is already closed, preventing unnecessary operations. (queue_service.py)

Batch Message Sending Improvements

• Refactored batch message sending to send all messages concurrently and handle errors more robustly, providing clear error logging and raising a specific exception on failure. (queue_service.py)