README.md

🎵 Consonance

Consonance is a robust event processing system built in Go that demonstrates the integration of Apache Kafka and Temporal for reliable, scalable event processing. It provides a framework for consuming, processing, and producing events with built-in workflow management and error handling.

🏗️ Architecture

Consonance combines several powerful technologies to create a reliable event processing pipeline:

🔋 Core Components

• 🚀 Apache Kafka: Handles event streaming and provides persistent message queues
• ⚙️ Temporal: Manages workflow orchestration and provides reliable task execution
• 🌐 Go API Server: Provides HTTP endpoints for workflow management

🔄 Event Flow

1. Raw events are published to example.raw.events Kafka topic
2. Temporal workflows coordinate the processing of events in configurable batches
3. Processed events are published to example.processed.events
4. Any processing errors are captured in the example.errors topic

📋 Prerequisites

• 🔧 Go 1.23.6 or later
• 🐳 Docker and Docker Compose
• ⚒️ Make (optional, but recommended)

🚀 Getting Started

1. Clone the repository:

   git clone https://github.com/yourusername/consonance.git
   cd consonance

2. Start the infrastructure:

   make start-infra

   This will start Kafka, Temporal, and their dependencies.

3. Start the application:

   make start-app

The following services will be available:

• 🌐 API Server: http://localhost:8082
• 📊 Kafka UI: http://localhost:8081
• ⚡ Temporal UI: http://localhost:8080

🔌 API Endpoints

💓 Health Check

GET /health

🔄 Workflow Management

POST /api/v1/workflows
Content-Type: application/json

{
    "batch_size": 10,
    "processing_interval": "30s"
}

GET /api/v1/workflows/{workflowID}

👩‍💻 Development

📁 Project Structure

.
├── docker/
│   └── docker-compose.yml    # Infrastructure setup
├── internal/
│   ├── api/                 # HTTP API implementation
│   ├── kafka/              # Kafka initialization and utilities
│   └── orchestration/      # Temporal workflows and activities
├── scripts/                # Utility scripts
└── main.go                # Application entry point

🔨 Making Changes

1. Format your code:

   go fmt ./...

2. Run tests:

   go test ./...

⚡ Available Make Commands

• make start-infra: Start infrastructure components
• make start-app: Start the application
• make logs: View logs from all services
• make down: Stop all services
• make clean: Clean up everything including volumes

⚙️ Configuration

Configuration is managed through YAML files:

• kafka.yml: Kafka topic configuration
• docker-compose.yml: Infrastructure service configuration

🤝 Contributing

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add some amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📜 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• ⚡ Temporal for workflow orchestration
• 🚀 Apache Kafka for event streaming
• 🌐 Chi Router for HTTP routing