Skip to content

v1.1.1 - Redis Pub/Sub Sharding & VU 1000 Load Test

Latest

Choose a tag to compare

@mangowhoiscloud mangowhoiscloud released this 27 Jan 21:42
· 18 commits to develop since this release

๐Ÿš€ Highlights

  • Redis Pub/Sub ์ฑ„๋„ ์ƒค๋”ฉ: user_id ํ•ด์‹œ ๊ธฐ๋ฐ˜ ์ฑ„๋„ ๋ถ„์‚ฐ์œผ๋กœ Hot Key ๋ฌธ์ œ ํ•ด๊ฒฐ
  • VU 500-1000 ๋ถ€ํ•˜ ํ…Œ์ŠคํŠธ ์™„๋ฃŒ: OpenAI Tier 4 ํ™˜๊ฒฝ์—์„œ VU 900๊นŒ์ง€ 99.7% ์„ฑ๊ณต๋ฅ  ๋‹ฌ์„ฑ
  • KEDA ScaledObject ์ตœ์ ํ™”: minReplicas 1โ†’2 (Cold Start ๋ฐฉ์ง€), maxReplicas 3โ†’5 (์ฒ˜๋ฆฌ๋Ÿ‰ ํ™•์žฅ)

๐Ÿ“Š Performance Results

VU ์„ฑ๊ณต๋ฅ  ์‹คํŒจ E2E P95 Throughput Worker Restart
500 99.7% 4 92.3s 351.9 req/m No
600 99.7% 4 108.3s 351.9 req/m No
700 99.2% 11 122.3s 329.1 req/m No
800 99.7% 4 144.6s 367.3 req/m No
900 99.7% 4 149.6s 405.5 req/m No
1000 97.8% 33 173.3s 373.4 req/m Yes

๊ถŒ์žฅ ์šด์˜ ๋ฒ”์œ„: VU 50-900 (99%+ ์„ฑ๊ณต๋ฅ )


โœจ What's New

Added

  • Redis Pub/Sub ์ฑ„๋„ ์ƒค๋”ฉ

    • sse:events:{user_id} โ†’ sse:events:{hash(user_id) % 8}
    • Hot Key ๋ฐฉ์ง€๋กœ Pub/Sub ์ฒ˜๋ฆฌ๋Ÿ‰ ๊ฐœ์„ 
  • VU 500-1000 ๋ถ€ํ•˜ ํ…Œ์ŠคํŠธ ๋ฆฌํฌํŠธ

    • docs/blogs/tests/2026-01-27-scan-load-test-vu800.md
    • docs/blogs/tests/2026-01-27-scan-load-test-vu900.md
    • docs/blogs/tests/2026-01-27-scan-load-test-vu1000-final.md

Changed

  • KEDA ScaledObject ์ตœ์ ํ™” (scan-worker)
    • minReplicas: 1 โ†’ 2 (Cold Start ๋ฐฉ์ง€)
    • maxReplicas: 3 โ†’ 5 (์ฒ˜๋ฆฌ๋Ÿ‰ 67% ์ฆ๊ฐ€)
    • Multi-Node ๋ถ„์‚ฐ: k8s-worker-ai + k8s-worker-ai-2

Fixed

  • Cold Start ์žฅ์•  ๋ฐฉ์ง€
    • minReplicas 2๋กœ ์„ค์ •ํ•˜์—ฌ VU 800+ ๋ถ€ํ•˜ ์‹œ Cascading Failure ๋ฐฉ์ง€
    • ์‹คํŒจ์œจ 37.7% ๊ฐ์†Œ (VU 1000 ๊ธฐ์ค€ 53๊ฑด โ†’ 33๊ฑด)

๐Ÿ” Key Findings

๋ณ‘๋ชฉ ๋ถ„์„

Priority Component Status
1 Celery Probe โŒ I/O-bound ์ƒํ™ฉ ์ทจ์•ฝ
2 Worker Concurrency โš ๏ธ 5 workers ร— 4 = 20 ๋™์‹œ ์ฒ˜๋ฆฌ
3 Node CPU/Memory โœ… ์—ฌ์œ  (60%/46%)
4 OpenAI TPM โœ… 61% ์‚ฌ์šฉ (39% headroom)

๊ถŒ์žฅ ์กฐ์น˜

  • Short-term: Probe timeout 30s๋กœ ์ฆ๊ฐ€
  • Medium-term: maxReplicas 7-8, Worker Node ์ถ”๊ฐ€
  • Long-term: HTTP Health Endpoint๋กœ ์ „ํ™˜

๐Ÿ“š Documentation


Full Changelog: v1.1.0...v1.1.1