Skip to content

Initialize worker cache before first request#670

Merged
hansott merged 2 commits into
mainfrom
agent/sync-worker-cache-before-first-request
Jul 24, 2026
Merged

Initialize worker cache before first request#670
hansott merged 2 commits into
mainfrom
agent/sync-worker-cache-before-first-request

Conversation

@teta2k

@teta2k teta2k commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

  • synchronize each worker process's local cache with the shared background process before a security check while that cache is uninitialized
  • serialize cache initialization and periodic synchronization-thread startup within each worker process
  • preserve fail-open behavior when IPC fails or the background process has not fetched configuration yet
  • add focused coverage for synchronization ordering, failures, concurrent requests, and existing-thread handling
  • mark manually constructed test configurations as already loaded when their policy is the subject of the test

Root cause

The Python agent has three relevant local components:

  1. a shared Aikido background process that fetches configuration from Core
  2. a separate process-local ThreadCache in every application worker process
  3. one periodic synchronization thread in every worker process that exchanges data with the background process every 15 seconds

Previously, the first HTTP request handled by a worker process could use its local cache before configuration had been copied from the background process. Starting the periodic synchronization thread did not guarantee that configuration was already available: the background process might not have fetched it yet, or the bounded IPC operation might not return it.

Worker-thread existence and local-cache readiness are therefore independent. The loader now synchronizes an uninitialized local cache immediately before checking whether the periodic thread already exists. A process-local lock serializes this operation so concurrent request threads cannot synchronize simultaneously or start duplicate periodic threads.

Impact

When a request reaches a worker process whose local cache has not received configuration, the agent performs a bounded synchronous IPC exchange with the shared background process instead of waiting for the next periodic synchronization. Once configuration has been received, normal requests do not perform this additional IPC operation.

If IPC fails or the background process still has no configuration, the request remains fail-open and the periodic synchronization thread continues its normal 15-second loop.

Validation

  • four-worker Gunicorn Compose reproduction test-allowed-ip-for-routes: passed all initial-policy and runtime-update phases, 1/1 passed in 208.76s using Python's established 100-second configuration propagation delay
  • four-worker Gunicorn A/B test after Core confirmed configuration delivery:
    • latest main: 28 of 100 requests from a non-allowed IP incorrectly returned 200
    • this PR: all 100 requests returned the required 403
  • focused regression coverage fails on main and passes with this patch
  • Windows Python 3.12 focused regression suite: 67 passed
  • pytest aikido_zen/thread aikido_zen/middleware -q — 33 passed
  • Black formatting check passed
  • GitHub QA comparison using one Gunicorn worker with four request threads:
    • main: 26 passed, 8 skipped, 0 failed
    • this PR: 26 passed, 8 skipped, 0 failed
  • four-worker Gunicorn Compose test test-user-rate-limiting-1-minute-without-x-forwarded-for passed in 18.64s

Summary by Aikido

Security Issues: 0 🔍 Quality Issues: 7 Resolved Issues: 0

⚡ Enhancements

  • Synchronized and initialized worker local cache before handling first request
  • Exposed is_config_loaded helper and ensured loader invoked during cache access

🐛 Bugfixes

  • Preserved fail-open behavior when IPC failed or configuration remained unavailable

🔧 Refactors

  • Serialized cache initialization and periodic-thread startup with a process-local lock
  • Adjusted end-to-end workflow to background processes and reordered startup commands

More info

@teta2k
teta2k marked this pull request as ready for review July 23, 2026 11:21
Comment thread aikido_zen/thread/process_worker.py Outdated
Comment thread aikido_zen/thread/thread_cache.py Outdated
Comment thread aikido_zen/thread/process_worker_loader.py Outdated
Comment thread aikido_zen/thread/process_worker_loader.py Outdated
@teta2k
teta2k force-pushed the agent/sync-worker-cache-before-first-request branch from 69d3d32 to 33070b6 Compare July 23, 2026 18:00
Comment thread aikido_zen/thread/process_worker_loader.py Outdated
@teta2k
teta2k force-pushed the agent/sync-worker-cache-before-first-request branch from 33070b6 to 8afb1c2 Compare July 23, 2026 19:11
Comment thread aikido_zen/thread/thread_cache.py
Comment thread aikido_zen/thread/process_worker_loader.py
@teta2k
teta2k force-pushed the agent/sync-worker-cache-before-first-request branch from 8afb1c2 to 5c75f9e Compare July 23, 2026 19:46
Comment thread aikido_zen/thread/process_worker_loader.py
@teta2k
teta2k force-pushed the agent/sync-worker-cache-before-first-request branch from 3baadca to e0f1cc1 Compare July 23, 2026 20:29
Comment thread aikido_zen/thread/thread_cache.py
Comment thread aikido_zen/thread/process_worker_loader.py
Comment thread aikido_zen/thread/process_worker_loader.py
Comment thread aikido_zen/thread/process_worker_loader.py Outdated
Co-authored-by: Hans Ott <hansott@hotmail.be>
Comment thread aikido_zen/thread/process_worker_loader.py
@hansott
hansott merged commit 87b4097 into main Jul 24, 2026
176 of 180 checks passed
@hansott
hansott deleted the agent/sync-worker-cache-before-first-request branch July 24, 2026 09:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants