Add support for request/response pattern

[mcollina]

Architecture Overview

The solution implements a bidirectional communication channel using Kafka topics:

• Request Topic: A single topic where all clients publish their requests
• Response Topics: Individual topics for each client (or client group)
• Correlation IDs: Unique identifiers linking requests with their corresponding responses
• Timeout Mechanism: Handling cases where responses aren't received

Message Structures

Request Message

The request message contains:

• A unique correlation ID to match requests with responses
• The name of the response topic where the client expects the response
• A timestamp for tracking and debugging
• The actual request payload

Response Message

The response message contains:

• The same correlation ID from the original request
• A timestamp for tracking and debugging
• The response payload
• A status indicator (SUCCESS, ERROR, TIMEOUT, etc.)

Implementation Details

Client Side Logic

The client side implementation should handle:

1. Generating unique correlation IDs for each request
2. Creating and managing a dedicated response topic
3. Sending requests to the request topic
4. Consuming responses from the response topic
5. Matching responses to pending requests using correlation IDs
6. Implementing timeouts for requests without responses
7. Providing a clean API for synchronous-style requests over the asynchronous Kafka infrastructure

Server Side Logic

The server side implementation should handle:

1. Consuming requests from the request topic
2. Processing each request through an application-specific handler
3. Creating appropriate response messages (success or error)
4. Sending responses to the client-specific response topics
5. Managing errors and ensuring responses are always sent
6. Implementing idempotency checks to handle duplicate requests

Scaling Considerations

Topic Partitioning

• The request topic should be partitioned to allow parallel processing
• Each server instance can consume from a subset of partitions using Kafka's consumer groups

Response Topic Management

Two approaches:

1. One response topic per client

   • Pros: Simple, direct routing, easier debugging
   • Cons: Topic proliferation, management overhead
   • Best for: Low to moderate client counts

2. Shared response topics with client IDs as message keys

   • Pros: Fewer topics, better for large numbers of clients
   • Cons: More complex routing, potential for "noisy neighbor" issues
   • Best for: High client counts, cloud environments

Load Balancing

• Server instances can be scaled horizontally
• Kafka's consumer group mechanism handles load distribution
• Consider sticky partitioning for related requests

Handling Edge Cases

Server Failures

• Implement acknowledgment when a server picks up a request
• If no acknowledgment, client can retry with the same correlation ID
• Consider idempotent request handlers

Client Failures

• Server can implement response TTL (Time-To-Live)
• Responses are discarded after TTL expires
• Consider periodic cleanup of orphaned responses

Duplicate Requests

• Server maintains a cache of recently processed correlation IDs
• Duplicates can be detected and handled appropriately
• Consider expiring cache entries after a reasonable period

Network Issues

• Implement retry logic with exponential backoff
• Consider circuit breakers for persistent failures
• Monitor and alert on communication patterns