Kafka-Backed Local Read Replica

Companion code for the article "Can Kafka Replace Redis for Cache Synchronization? A Production Answer" on codewithkyryl.dev — a compacted Kafka topic broadcast to a thread-safe local cache on every instance, with a readiness gate that keeps traffic and scheduled jobs off until it's fully replayed the topic. Runs entirely on docker compose, Kafka in KRaft mode (no ZooKeeper), with a kafka-ui to watch the consumer groups do their thing — no Kubernetes or minikube required.

This repo started from a generic Spring Boot template and has been trimmed down to only what this demo needs — no database, no HTTP client stack, no API docs generator. Two Gradle modules: readiness-gate (a standalone library — "don't accept traffic or run scheduled jobs until you've fully caught up") and app (the demo, which depends on it).

Stack

Concern	Choice
Language / runtime	Java 25 (LTS), auto-provisioned via Gradle toolchains
Framework	Spring Boot 4.1 (Spring Framework 7)
Build	Gradle 9.6 (Kotlin DSL)
Messaging	Spring for Apache Kafka, KRaft-mode broker, kafka-ui
Errors	RFC 9457 ProblemDetail responses
Testing	JUnit 5 + Testcontainers (real Kafka, KRaft mode)
Ops	Actuator (health/info/metrics/prometheus), graceful shutdown, readiness gating
Tooling	Spotless (Palantir format), JaCoCo, GitHub Actions CI, Renovate, Docker

Prerequisites

• JDK 25 — or nothing at all; Gradle downloads the right JDK via the toolchain resolver.
• Docker (with Compose) — for local infra, containerized builds, and Testcontainers-based tests.

Quick start

# Run the app — spring-boot-docker-compose starts Kafka (+ kafka-ui, + a one-shot job that
# creates the compacted topic and seeds it) for you
make run          # or: ./gradlew :app:bootRun

# Run the full test suite
make test         # or: ./gradlew test

Once running:

• API base: http://localhost:8080
• Cache endpoint: http://localhost:8080/api/cache
• Health: http://localhost:8080/actuator/health (readiness group: /actuator/health/readiness)
• kafka-ui: http://localhost:8090

Run make help to see all available commands.

Configuration

Configuration lives in app/src/main/resources/application.yml (one file, no profile split — this demo doesn't need one) and reads from environment variables with sensible local defaults:

Variable	Default	Purpose
APP_NAME	local-cache-demo	Application name
SERVER_PORT	8080	HTTP port
KAFKA_BOOTSTRAP_SERVERS	localhost:9092	Kafka broker(s)
CONFIG_TOPIC	local-cache-config	Compacted topic the app consumes
CONSUMER_GROUP_PREFIX	local-cache-demo	Human-picked, set in docker-compose.yml — see below

Spring Boot 4.1 ships no Kafka autoconfiguration yet (no spring.kafka.* binding), so the consumer is wired by hand in app/src/main/java/com/example/company/config/KafkaConfig.java.

logging.level in application.yml also shows how to override one specific logger (here, ReadinessGateAspect, set to DEBUG) without dropping the whole app to debug-level noise — a named logger always wins over a broader package prefix.

Project layout

Two Gradle modules — readiness-gate has no dependency on app; it uses Kafka's consumer API to compare processed offsets with the end offsets captured on partition assignment.

readiness-gate/src/main/java/com/example/company/readinessgate
├── ConfigConsumer.java                    # marker annotation for gated listener methods
├── ConsumptionTracker.java                # coordinates progress for registered consumers
├── ConsumerReplayProgress.java            # assignment and replay state for one consumer
├── PartitionReplayProgress.java           # position, fixed target, and error for one partition
├── ConfigConsumerAspect.java              # @Around, reports every invocation to the tracker
├── ConfigConsumerRegistrar.java           # pre-registers @ConfigConsumer methods at startup
├── ConfigConsumptionHealthIndicator.java  # exposes tracker state to the "readiness" health group
├── ReadinessGated.java                    # marker annotation: skip this method until fully consumed
├── ReadinessGateAspect.java               # @Around, skips @ReadinessGated methods until then
└── ReadinessGateAutoConfiguration.java    # self-registering, works from any base package

app/src/main/java/com/example/company
├── Application.java          # entry point
├── cache/                    # LocalConfigCache (thread-safe) + CacheEntry
├── kafka/                    # ConfigTopicListener + shared consumer id; applies records and tombstones
├── job/                      # CacheSnapshotJob — dummy @Scheduled job, gated on readiness
├── config/                   # KafkaConfig, GlobalExceptionHandler, JobLoggingAspect
├── controller/                # CacheController (/api/cache)
└── exception/                 # ApplicationException

Testing

Integration tests extend AbstractIntegrationTest (app/src/test/...), which boots the full context against a real Kafka broker (KRaft mode) started by Testcontainers — no local infra required. Docker must be running.

• ConfigTopicCacheIntegrationTest drives the whole demo end to end: produce onto the topic, wait for the readiness gate to open, read it back through /api/cache, then verify a tombstone evicts the key.
• ConsumptionTrackerTest (readiness-gate module) unit-tests the gate itself: stays DOWN until every assigned partition reaches its captured end offset, including empty topics, processing failures, revoked partitions, and the fixed-target behavior for records appended during replay.
• ReadinessGateAspectTest (readiness-gate module) unit-tests the @ReadinessGated aspect via AspectJProxyFactory — no Spring context needed for it either.
• Coverage — ./gradlew test runs JaCoCo per module; reports land in */build/reports/jacoco/.

Containerize

Build a single image (multi-module: docker build runs ./gradlew :app:bootJar, which transitively builds readiness-gate too):

make image                       # docker build -t local-cache-demo .
docker run --rm -p 8080:8080 local-cache-demo

Or run the whole stack (app + Kafka + kafka-ui) with Compose:

cp .env.example .env             # tweak values as needed
make up                          # docker compose --env-file .env.example --profile full up --build -d
make down                        # stop it

make up uses .env when present and falls back to .env.example. The app service sits behind the Compose full profile, so ./gradlew :app:bootRun still only auto-starts kafka, kafka-ui, and kafka-topic-init — not a second copy of the app itself.

The multi-stage Dockerfile produces a layered, non-root image on a slim JRE.

Kafka-backed local cache demo

This is the part the article is actually about: can Kafka replace Redis for broadcasting cache updates to every instance? The answer this repo demonstrates — yes, with a compacted topic, one consumer group per instance, and a readiness gate — end to end, with a single docker compose command.

Run it

cp .env.example .env    # tweak values as needed
make up                 # builds the image, starts kafka + kafka-ui + app
make urls                # prints the URLs below

Talk to it:

curl localhost:8080/api/cache
curl localhost:8080/api/cache/feature.dark-mode

Both should respond as soon as make up finishes — the topic is created and seeded by kafka-topic-init (a one-shot Compose service) before the app service is even allowed to start (depends_on: condition: service_completed_successfully), so there's no "wait a bit and retry" step. The readiness endpoint remains DOWN until the app has completed its own initial replay; Docker Compose reports the app as unhealthy until that endpoint turns UP.

Open http://localhost:8090 (kafka-ui) and check:

• Topics → local-cache-config — cleanup.policy=compact, 3 partitions, 5 seeded keys.
• Consumers — one consumer group for the running app container, holding all 3 partitions (not split — that's the broadcast, not load-balance, in action).

How the consumer group name is built

An instance needs a distinct Kafka consumer group to get the entire topic broadcast to it, instead of Kafka load-balancing partitions across instances the way a shared group normally would (see the article for why). The name is assembled in two places:

1. CONSUMER_GROUP_PREFIX (docker-compose.yml, app service) — a human-picked value, the piece you'd actually want to control per-deployment.
2. ${random.uuid} — generated by Spring for each application instance.

Spring's property placeholder resolution combines them in app/src/main/resources/application.yml: ${CONSUMER_GROUP_PREFIX}-${random.uuid} becomes the real group-id passed to the Kafka consumer (config.KafkaConfig). Each instance therefore consumes the full topic independently. A restart receives a new group ID and replays the compacted topic from the beginning.

Scale the app service and you'll see a new, distinct consumer group appear in kafka-ui for every new container — and it stays behind after the container is gone, same as the "dead consumer groups" the article talks about. That's expected, not a leak to chase down. Note that scaling needs the fixed host port removed first (Compose can't bind 8080:8080 for more than one replica) — drop the ports: line under the app service, then:

docker compose --env-file .env --profile full up -d --scale app=3

Deleting a key (tombstones)

The topic is compacted, so deletes use Kafka's tombstone convention: a record with the same key and a null value. ConfigTopicListener (app/src/main/java/com/example/company/kafka/) checks for exactly that and evicts the key from the local cache instead of caching a null:

docker compose exec kafka bash -c '
  kafka-console-producer --bootstrap-server localhost:9092 \
    --topic local-cache-config --property "parse.key=true" --property "key.separator==" \
    --property "null.marker=NULL" <<< "feature.dark-mode=NULL"
'

curl localhost:8080/api/cache/feature.dark-mode   # 404 once the tombstone's been consumed

Tear down

make down            # or: docker compose --profile full down

Add -v (docker compose --profile full down -v) to also drop the Kafka volume and start completely clean next time.

CI & dependency updates

• GitHub Actions (.github/workflows/ci.yml) checks formatting, runs ./gradlew build on every push and PR, and uploads the test and coverage reports.
• Renovate (renovate.json) opens grouped dependency-update PRs and auto-merges safe minor/patch bumps.

License

MIT