distributed-chat

A real-time, multi-instance workspace messaging system built with Spring Boot, Apache Kafka, and WebSocket/STOMP. Think a minimal slice of Slack or Discord — not a production product, but a deliberate learning project that demonstrates how a horizontally scalable, event-driven backend actually works.

The core problem this project solves: Spring's built-in STOMP broker is in-memory. The moment you run two server instances, a message sent to one never reaches a client connected to the other. Kafka is the shared bus that bridges them — and the way it does so comes down to a single configuration decision about consumer group IDs.

---

Architecture

┌─────────────┐                              ┌─────────────┐
│   Client A  │                              │   Client B  │
│ :8080/ws    │                              │ :8081/ws    │
└──────┬──────┘                              └──────┬──────┘
       │ STOMP SEND                                 │ subscribes
       ▼                                            ▼
┌──────────────────────────┐        ┌──────────────────────────┐
│  Instance A — :8080      │        │  Instance B — :8081      │
│                          │        │                          │
│  AuthChannelInterceptor  │        │  AuthChannelInterceptor  │
│  ChatController          │        │  ChatController          │
│  ChatService             │        │  ChatService             │
│  MessageRepository       │        │  MessageRepository       │
│                          │        │                          │
│  group: chat-service-    │        │  group: chat-service-    │
│         {uuid-A}         │        │         {uuid-B}         │
└───────────┬──────────────┘        └──────────────┬───────────┘
            │ publish                    consume │  │ consume
            │       ┌────────────────┐           │  │
            └──────►│ Kafka          │◄──────────┘  │
                    │ chat.messages  │──────────────┘
                    └────────────────┘
            │ save
            ▼
┌─────────────────┐
│   PostgreSQL    │
│   messages      │
│   channels      │
│   users         │
│   workspaces    │
└─────────────────┘

Why two different consumer group IDs?

Kafka delivers each message to exactly one consumer per consumer group. If both instances shared the same group ID, Kafka would treat them as competing workers and only one would receive each message — silently breaking delivery for half your clients, with no error.

By setting group-id: chat-service-${random.uuid}, each JVM generates a unique group ID at startup. Kafka treats them as completely independent consumers. Both receive every message. Both broadcast to their own locally connected clients. That is fan-out.

This is the single most important line in this project:

spring:
  kafka:
    consumer:
      group-id: chat-service-${random.uuid}

---

Message flow — step by step

1.  Client A sends STOMP SEND to /app/chat.send
            │
2.  AuthChannelInterceptor validates JWT
    (attached at CONNECT time, not per-message)
            │
3.  ChatController.handleIncomingMessage()
    receives ChatMessageRequest + authenticated Principal
            │
4.  ChatService.processIncomingMessage()
    ├── looks up Channel by ID
    ├── resolves sender from Principal (not from payload)
    ├── saves Message to PostgreSQL   ◄── happens FIRST
    └── publishes ChatMessageEvent to Kafka
                   │
5.  Kafka broker delivers to ALL consumer groups
    ├── Instance A consumer receives it
    │        └── broadcasts to /topic/channels/{id}/messages
    │                 └── Client A receives MESSAGE frame
    │
    └── Instance B consumer receives it
             └── broadcasts to /topic/channels/{id}/messages
                      └── Client B receives MESSAGE frame

Why save before publish? If the save fails, no broadcast happens — clients never see a message that does not exist in history. If publish happened first, clients could see a message that was never persisted.

---

Request types at a glance

STOMP (real-time)

Client sends to	What happens
/app/chat.send	Message saved to Postgres, published to Kafka, broadcast to all instances
/app/chat.typing	Typing indicator broadcast directly — no Kafka, best-effort delivery

Client subscribes to	What arrives
/topic/channels/{id}/messages	Chat messages for that channel
/topic/channels/{id}/typing	Typing indicators for that channel
/topic/presence	User online/offline events
/user/queue/errors	Validation or lookup errors, sent only to the requesting client

REST

Method	Path	Description
GET	/api/channels/{id}/messages?page=0&size=20	Paginated message history, newest first
GET	/actuator/health	Health check
GET	/actuator/metrics/chat.messages.published	Kafka publish counter
GET	/actuator/metrics/chat.messages.broadcast	STOMP broadcast counter
GET	/actuator/metrics/chat.messages.deduped	Kafka duplicate delivery counter

---

Tech stack

Concern	Technology	Why
Language	Java 21	Records, modern APIs
Framework	Spring Boot 3.5	Web, JPA, WebSocket, Validation, Actuator
Real-time messaging	WebSocket + STOMP	Open persistent connection, pub/sub semantics
Cross-instance fan-out	Apache Kafka (KRaft)	Durable shared message bus, no ZooKeeper needed
Persistence	PostgreSQL + Spring Data JPA	Reliable, queryable message history
Schema management	Flyway	Versioned migrations, ddl-auto: validate
Authentication	JJWT 0.13 + ChannelInterceptor	JWT verified at CONNECT frame only
Metrics	Micrometer + Spring Boot Actuator	Custom counters, health endpoint
Tests	JUnit 5 + Mockito	Unit tests for service and listener logic
Infrastructure	Docker Compose	Postgres + Kafka, single command startup

---

Prerequisites

• Java 21+
• Docker Desktop
• Maven (the wrapper ./mvnw is included)

---

Running it locally

1. Clone the repo

git clone https://github.com/Maheesh09/chatworkspace.git
cd chatworkspace/chat

2. Create your local secrets file

This file is gitignored. Create it at src/main/resources/application-local.yml:

app:
  jwt:
    secret: your-secret-key-must-be-at-least-32-characters-long

The secret must be at least 32 characters — HS256 enforces a minimum key length.

3. Start Postgres and Kafka

docker compose up -d

• PostgreSQL available at localhost:5433
• Kafka available at localhost:9092

Wait for Kafka to finish initializing:

docker compose logs -f kafka
# Look for: [BrokerServer id=1] Transition from STARTING to STARTED

4. Build the jar

./mvnw clean package -DskipTests

5. Run two instances

Open two terminal windows in the project root:

# Terminal 1
java -jar target/chat-0.0.1-SNAPSHOT.jar --server.port=8080

# Terminal 2
java -jar target/chat-0.0.1-SNAPSHOT.jar --server.port=8081

Each instance prints its unique Kafka consumer group ID at startup. Confirm they differ:

# Instance A log:
group.id = chat-service-43647a0f-c969-466c-9990-a2bfe28e286f

# Instance B log:
group.id = chat-service-f4fe5659-60b9-48fc-93cc-9be67a472b34

Different UUIDs = independent Kafka consumers = fan-out is live.

---

Proving fan-out works

Use a WebSocket client with raw frame support (Postman works with Binary → Hexadecimal mode).

1. Tab A — connect to ws://localhost:8080/ws
2. Tab B — connect to ws://localhost:8081/ws
3. On both tabs: send CONNECT (with JWT in Authorization header), then SUBSCRIBE to /topic/channels/1/messages
4. From Tab A only: send a SEND frame to /app/chat.send
5. Watch Tab B's response pane — the same message arrives, ~100ms later, even though Tab B is connected to a completely separate JVM and never sent anything

That ~100ms is the Kafka round trip: publish → broker → consume → broadcast.

---

Seeding test data

There is no registration endpoint. Seed directly into the database:

docker exec -it chat-postgres psql -U chat_user -d chatdb

INSERT INTO workspaces (name, created_at) VALUES ('Test Workspace', now());
INSERT INTO users (username, display_name, created_at) VALUES ('alice', 'Alice', now());
INSERT INTO channels (name, workspace_id, created_at) VALUES ('general', 1, now());

Use channelId: 1 and sub: "1" in your JWT and STOMP frames.

---

Generating a test JWT

The project has no /auth/login endpoint. Use any JWT tool with these settings:

• Algorithm: HS256
• Secret: the value from your application-local.yml
• Claim: sub = "1" (the user ID as a string)
• Expiry: 24 hours from now

Pass it in the STOMP CONNECT frame as: Authorization: Bearer <token>

---

Key design decisions

JWT at CONNECT, not per-message

AuthChannelInterceptor.preSend() fires on the STOMP CONNECT frame. It validates the JWT and attaches a StompPrincipal(userId) to the session. Spring carries this Principal for the life of the connection. Every subsequent SEND on that session is already authenticated — the client never sends the token again, and there is no way to claim a different identity mid-session.

No senderId in the request payload

ChatMessageRequest has no senderId field. The sender is resolved from the verified StompPrincipal — not from anything the client says in the message body. A client crafting a SEND frame to impersonate another user has no field to exploit.

Typing indicators bypass Kafka

Kafka adds ~50–150ms latency even on localhost. For chat messages this is fine. For typing indicators, that latency is noticeable and the UX breaks. Typing indicators broadcast directly via SimpMessagingTemplate, skipping Kafka entirely. If a client is on a different instance, they may miss the indicator — that is acceptable. Missing "Alice is typing…" for 100ms is a cosmetic glitch. Missing a chat message is data loss. Different problems, different tools.

ddl-auto: validate with Flyway owning the schema

Hibernate checks that every @Entity matches the real database tables at startup, but never creates, alters, or drops anything. V1__create_initial_schema.sql is the single source of truth for schema shape. If you change an entity without updating the migration, startup fails immediately and loudly rather than silently drifting.

TIMESTAMPTZ not TIMESTAMP

Hibernate 6 maps java.time.Instant to TIMESTAMP WITH TIME ZONE. Using plain TIMESTAMP in the SQL migration would cause ddl-auto: validate to fail at startup with a column type mismatch. A version-specific detail that bites many people migrating from Hibernate 5.

Idempotent Kafka consumption

Kafka's at-least-once guarantee means a message can be redelivered after a consumer crash. A bounded in-memory LRU set (last 1000 message IDs) in ChatMessageListener detects and skips duplicates. If the ID was already processed, the listener logs a warning and returns without broadcasting.

Known limitation: this set resets on restart and is per-instance only. In production, Redis with a short TTL would be the correct solution.

Presence is per-instance only

SimpUserRegistry tracks STOMP sessions on this JVM only. Instance A has no knowledge of who is connected to Instance B. Presence events fire correctly for clients on the same instance.

Known limitation: Redis-backed shared presence state would fix this. Deliberately out of scope — this project demonstrates Kafka fan-out, not full presence infrastructure.

Save before publish — always

ChatService.processIncomingMessage saves the message to Postgres before calling messagePublisher.publish(). If the save fails, no Kafka event is published and no client sees the message. This ordering is a correctness requirement, not a preference.

GenerationType.IDENTITY over SEQUENCE

IDENTITY makes JDBC batch-inserts impossible since Hibernate needs each generated ID back immediately. SEQUENCE supports batching. IDENTITY was chosen here because messages are created one at a time as real events happen — bulk insert is never needed, so the downside of IDENTITY never materializes.

---

Project structure

src/main/java/com/dca/chat/
├── config/               WebSocketConfig, KafkaConfig
├── domain/entity/        User, Workspace, Channel, Message
├── dto/                  ChatMessageRequest, ChatMessageEvent,
│                         MessageResponse, TypingIndicatorRequest,
│                         TypingIndicatorEvent, PresenceEvent, ErrorResponse
├── exception/            ChannelNotFoundException, UserNotFoundException
├── messaging/
│   ├── listener/         ChatMessageListener, PresenceEventListener
│   └── publisher/        MessagePublisher (interface),
│                         KafkaChatMessagePublisher
├── repository/           UserRepository, WorkspaceRepository,
│                         ChannelRepository, MessageRepository
├── security/             AuthChannelInterceptor, JwtService, StompPrincipal
├── service/              ChatService, PresenceService
├── stomp/                ChatController
└── web/                  ChannelHistoryController

src/main/resources/
├── application.yml                   main configuration
├── application-local.yml             gitignored — JWT secret lives here
└── db/migration/
    └── V1__create_initial_schema.sql Flyway migration

src/test/java/com/dca/chat/
├── ChatApplicationTests              full Spring context load
├── service/
│   └── ChatServiceTest               4 unit tests
└── messaging/listener/
    └── ChatMessageListenerTest       4 unit tests

---

Tests

No Testcontainers integration tests — the fan-out behavior was proven manually with two live JVM instances. Testcontainers would be the natural next step.

ChatServiceTest — 4 tests

Test	What it proves
processIncomingMessage_whenValidRequest_savesAndPublishes	Save and publish both called on the happy path
processIncomingMessage_whenChannelNotFound_throwsChannelNotFoundException	Exception thrown, save and publish never called
processIncomingMessage_whenUserNotFound_throwsUserNotFoundException	Exception thrown, save and publish never called
processIncomingMessage_publishesAfterSave	Published event carries the correct content

ChatMessageListenerTest — 4 tests

Test	What it proves
consumeChatMessage_whenNewMessage_broadcastsToSubscribers	New ID causes exactly one broadcast
consumeChatMessage_whenDuplicateMessage_doesNotBroadcast	Same ID twice causes only one broadcast
consumeChatMessage_whenDifferentMessages_broadcastsBoth	Two different IDs each broadcast once
consumeChatMessage_whenMessageNotSeen_doesNotSkip	Dedup is ID-specific, does not suppress other messages

Run all tests:

./mvnw clean test

Expected output: Tests run: 9, Failures: 0, Errors: 0, Skipped: 0

---

Observability

With the app running:

# Health
curl http://localhost:8080/actuator/health

# Message counters
curl http://localhost:8080/actuator/metrics/chat.messages.published
curl http://localhost:8080/actuator/metrics/chat.messages.broadcast
curl http://localhost:8080/actuator/metrics/chat.messages.deduped

Log correlation via MDC: every Kafka message processed includes messageId and channelId in the logging context automatically. All log lines during that message's processing carry these values.

Log format:

HH:mm:ss.SSS [thread] LEVEL logger [msgId=42 ch=1] - message text

---

What this project deliberately excludes

Excluded	Reason
User registration / login endpoint	Tokens generated manually for testing
Channel membership model	Any authenticated user can post to any channel
HTTP authentication on REST endpoints	STOMP is auth-gated; HTTP auth was out of scope
Redis	Needed for shared presence state and production-grade dedup; excluded to keep focus on Kafka fan-out
Testcontainers integration tests	Fan-out proven manually; Testcontainers is the natural next step
Real frontend	Postman is sufficient to demonstrate everything this project is actually about

---

Build roadmap (completed)

Step	What was built
1	Single-instance STOMP chat skeleton
2	Domain model + PostgreSQL + Flyway migrations
3	Service layer, DTOs, Bean Validation, STOMP error handling
4	JWT authentication at STOMP handshake via ChannelInterceptor
5	Kafka as single-instance pass-through, round-trip proven
6	Multi-instance fan-out proven across two JVM processes
7	Paginated REST history endpoint (newest-first, indexed)
8	Typing indicators (best-effort) + presence tracking
9	Idempotent Kafka consumption + malformed message error handling
10	Unit tests — ChatService and ChatMessageListener
11	Observability: MDC correlation IDs, Micrometer counters, Actuator

Step 6 is the milestone this project exists to demonstrate. Everything before it builds the foundation; everything after it makes it production-honest.

---