[Research] Chaos Testing & Resilient Validation

[kingazm]

• Research the topic
• Come up with a strategy for out app

Name of a suggested app: K6

[kingazm]

Please put some notes for us:) @avvvis

[kingazm]

+screnarios

[avvvis]

Below I propose a plan made with AI tools. Seems like a decent place to start, if anything is missing let me know :))

Chaos Testing Plan — JuCanEat Java Backend

What Is Chaos Testing & Why It Matters Here

Chaos testing deliberately injects failures into a running system to verify it degrades gracefully,
recovers predictably, and never corrupts data. This system has 5 external dependencies (PostgreSQL,
RabbitMQ, Keycloak, OpenAI, DeepL), async processing pipelines, and real-money food ordering flows —
making uncontrolled failure expensive.

---

1. Database Failure (PostgreSQL)

Tests:

• Kill PostgreSQL container mid-request (docker stop postgres_java_backend)
• Introduce network latency to port 5432 (tc netem delay 5000ms)
• Exhaust connection pool by opening max connections externally
• Kill DB during an active Flyway migration
  Why critical:
  Every API endpoint reads or writes PostgreSQL. There is no caching layer and no read replica.
  No circuit breaker (Resilience4j absent), so threads pile up waiting for connections until the
  app becomes unresponsive. Connection pool exhaustion is a realistic production scenario under load.

Verify:

• All endpoints return 503 (not 500 or hung connection)
• No partial writes leave daily_menu stuck in PROCESSING permanently
• Connection pool recovers when DB comes back — no manual restart needed
• Flyway does not corrupt schema on interrupted migration
  Risk: HIGH — no circuit breaker, no timeout on DB pool acquisition

---

2. Message Queue Failure (RabbitMQ)

Tests:

• Stop RabbitMQ after menu image upload (status = PROCESSING)
• Restart RabbitMQ with messages in-flight
• Flood queue with 100+ simultaneous menu uploads
• Disconnect/reconnect RabbitMQ repeatedly while listener is active
  Why critical:
  The entire menu image upload pipeline (POST /api/menus) is async and RabbitMQ-dependent.
  If the broker goes down after 202 Accepted is returned, the menu stays in PROCESSING forever.
  No polling fallback, no timeout to flip status to FAILED. This is a silent data corruption scenario.

Verify:

• Menus stuck in PROCESSING are detected and moved to FAILED after a timeout
• RabbitMQ reconnects automatically without app restart
• Dead-letter queue (menu.dlx) captures failed messages for replay
• SSE clients receive a failure notification when processing cannot complete
  Risk: HIGH — no processing timeout, no status cleanup job

---

3. Authentication Service Failure (Keycloak)

Tests:

• Stop Keycloak container entirely
• Introduce 3–10 second latency on port 8081
• Restart Keycloak while users are actively sending requests
• Corrupt Keycloak realm config and restart
  Why critical:
  All protected endpoints require JWT validation against Keycloak's JWKS endpoint. Spring Security
  caches the JWKS set so existing tokens may work briefly, but new logins and token refreshes fail.
  SaveUserFilter also queries Keycloak on first login — throws exceptions if Keycloak is down.

Verify:

• Previously-cached valid JWTs continue working during short Keycloak outages
• Expired tokens return 401, not 500
• SaveUserFilter fails gracefully without blocking the request pipeline
• System recovers fully when Keycloak restarts — no app restart required
  Risk: MEDIUM — JWKS caching provides some resilience, but no explicit fallback

---

4. AI Service Failure (OpenAI GPT-4O Mini)

Tests:

• Block outbound HTTPS to api.openai.com via firewall rule
• Return 429 Too Many Requests from a mock server
• Return malformed/empty JSON responses
• Simulate timeout (response delayed > 30 seconds)
  Why critical:
  ProdAIService has a 5-attempt retry loop but no explicit HTTP timeout. A slow OpenAI response
  blocks the MenuProcessingListener thread indefinitely. Because RabbitMQ listener threads are
  limited, a wave of slow responses can exhaust all listener threads, freezing menu processing for
  ALL restaurants — not just the one that triggered it.

Verify:

• A timeout (e.g., 15s per attempt) is enforced on all OpenAI HTTP calls
• Menu is marked FAILED (not stuck in PROCESSING) when all retries are exhausted
• SSE notification is sent to user on FAILED status
• Other restaurants' menus are not blocked by one slow upload
  Risk: HIGH — no HTTP timeout configured, thread starvation possible

---

5. Translation Service Failure (DeepL)

Tests:

• Block outbound traffic to api-free.deepl.com
• Return 403 Forbidden (quota exceeded) or 429 Too Many Requests
• Simulate partial failure — only some dish translations fail
• Simulate 5-second latency per translation call
  Why critical:
  A graceful fallback exists (source language name used on failure) — good. However, there is no
  batch translation: one HTTP call per dish means a 10-dish menu = 10 sequential HTTP calls.
  If DeepL is slow (not down), this causes significant latency per menu processing cycle.

Verify:

• All dishes appear in API response even when DeepL is fully down
• Missing translations use source-language names, not null or empty strings
• A 5-second DeepL latency per call does not cause overall menu processing to time out
• Warning logs are emitted and visible in Prometheus metrics
  Risk: LOW (fallback exists) / Latency risk: MEDIUM

---

6. Network Partition Between Containers

Tests:

• Partition Java backend from PostgreSQL only (other services up)
• Partition Java backend from RabbitMQ only
• Full partition: isolate Java backend from all containers
• Introduce 20–50% packet loss on internal Docker network
  Why critical:
  No explicit connection timeouts are configured anywhere in application properties. Under packet
  loss or intermittent connectivity, Spring's default timeouts (often minutes) cause requests to
  hang rather than fail fast. Kubernetes liveness probes will eventually kill the pod, but not
  before user-facing requests time out completely.

Verify:

• GET /health responds within 1 second even when external services are partitioned
• All external service calls fail within 5–10 seconds (explicit timeouts)
• Readiness probe (/actuator/health) correctly reports DOWN when DB is unreachable
• App recovers automatically after network restoration — no manual intervention
  Risk: HIGH — no explicit timeouts configured on any external calls

---

7. Scheduled Job Failures (MenuPublishingScheduler)

Tests:

• Run scheduler while PostgreSQL is unavailable
• Inject 1000 SCHEDULED menus and measure scheduler runtime
• Run scheduler concurrently from two app instances (simulate multi-pod K8s)
• Kill app mid-scheduler-run between menu status transitions
  Why critical:
  MenuPublishingScheduler runs daily at 0 5 * * * * and flips SCHEDULED → ACTIVE.
  In Kubernetes with multiple pods, both may attempt to activate the same menu simultaneously —
  producing duplicate ACTIVE menus for the same restaurant on the same date. No distributed
  locking (e.g., ShedLock) protects this scheduler.

Verify:

• Only one pod activates each menu (distributed lock or idempotent update query)
• DB failure during run logs an error and retries next cron tick — does not crash the app
• No menu is left in SCHEDULED permanently if activation partially fails
• Scheduler completes in under 1 minute even with large volumes of scheduled menus
  Risk: HIGH — no distributed locking, race condition in multi-pod deployment

---

8. SSE Connection Failures (Real-Time Notifications)

Tests:

• Drop client SSE connection mid-processing, then complete processing
• Open 500+ simultaneous SSE connections and kill the broker
• Process 50 menus simultaneously and verify all SSE events are delivered
• Reconnect SSE after server restart
  Why critical:
  SseEmitterService stores emitters in a ConcurrentHashMap keyed by user ID. If the client
  disconnects and reconnects, a new emitter is registered — but the result of already-in-progress
  processing may be sent to the old, dead emitter. The user never receives the notification and
  must manually poll for menu status. No persistence layer or replay mechanism exists for SSE events.

Verify:

• Dead emitters are cleaned up promptly (no memory leak)
• Users who reconnect SSE can poll for current menu status
• 500 concurrent SSE connections do not degrade API response times
• App does not crash when sending to a closed emitter
  Risk: MEDIUM — missing delivery guarantee, potential memory leak on dead emitters

---

9. Load & Stress Testing

Tests:

• 200 concurrent GET /api/restaurants for 5 minutes (baseline)
• 50 simultaneous menu image uploads
• 1000 rapid POST /api/users/favourites/{id} requests from same user
• Gradual ramp: 0 → 500 concurrent users over 10 minutes
  Why critical:
  No pagination exists on GET /api/restaurants, GET /api/facilities, or dish listings within
  menus. A single request could return thousands of objects with nested dishes, locations, and hours.
  Under load this causes N+1 query problems (Hibernate lazy loading) and large GC pressure.

Verify:

• API responses stay under 500ms at 100 concurrent users
• Memory usage stays stable (no GC spiral) under sustained load
• No database connection pool exhaustion (HikariCP pool metrics)
• GET /health stays responsive during peak load (separate thread pool)
  Risk: MEDIUM-HIGH — no pagination, potential N+1 queries

---

10. Data Integrity Under Concurrent Operations

Tests:

• Two owners simultaneously upload a menu for the same restaurant
• Delete a restaurant while its menu is being processed
• Assign same user as owner twice simultaneously
• Concurrent add/remove favorites for the same facility
  Why critical:
  No optimistic locking (@Version) exists on DailyMenu or Restaurant entities. Two owners
  making simultaneous changes cause last-write-wins data loss with no error raised. The PROCESSING
  status check in MenuService relies on a read-then-write pattern that is not atomic — a race
  window exists between the status check and the update.

Verify:

• Concurrent menu uploads for the same restaurant return 409 Conflict for the second request
• No PROCESSING menu duplicates are created under race conditions
• Deleting a restaurant while processing its menu moves the menu to FAILED cleanly
• All concurrent-write scenarios return correct HTTP status codes, not 500
  Risk: HIGH — no optimistic locking, non-atomic status check-and-set

---

Priority Matrix

#	Test Area	Severity	Likelihood	Current Mitigation	Priority
1	PostgreSQL failure	Critical	Medium	None (no circuit breaker)	P0
2	RabbitMQ failure	Critical	Medium	DLX configured	P0
6	Network partition / timeouts	Critical	High	None	P0
10	Concurrent data writes	High	High	Status check (not atomic)	P0
4	OpenAI timeout/failure	High	Medium	5-retry loop (no timeout)	P1
7	Scheduler race condition	High	High (K8s)	None	P1
3	Keycloak failure	Medium	Low	JWKS caching	P1
9	Load/stress	Medium	Medium	None	P2
8	SSE connection drops	Medium	High	None	P2
5	DeepL failure	Low	Medium	Fallback exists	P3

---

Recommended Fixes Before Chaos Testing

1. Add Resilience4j circuit breakers on OpenAI, DeepL, and Keycloak HTTP calls
2. Configure explicit HTTP timeouts on all WebClient/RestTemplate beans (OpenAI, DeepL)
3. Add @Version optimistic locking to DailyMenu and Restaurant entities
4. Add ShedLock to MenuPublishingScheduler for safe K8s multi-pod execution
5. Add a ProcessingTimeoutJob — scheduled task that moves stale PROCESSING menus to FAILED after N minutes
6. Add pagination to GET /api/restaurants and GET /api/facilities

---

Tooling Recommendations

Tool	Purpose
Chaos Monkey for Spring Boot	Random bean/latency injection in-process
Toxiproxy	Network latency, packet loss, connection limits
Gatling	Load & stress tests with ramp-up scenarios
Testcontainers	Integration tests with real PostgreSQL/RabbitMQ
WireMock	Mock OpenAI/DeepL with configurable failure modes
k6	HTTP-level load testing for API endpoints