Nginx and API Gateway

Canonical source: docs/08-runtime-devops/nginx-reference.md — this page is rendered from it on every publish; edit the source.

Nginx Configuration Reference

Overview

Nginx runs as a reverse proxy on port 4000, routing all API requests from the frontend to the appropriate backend service. It handles DNS resolution, timeout management, security headers, and SSE (Server-Sent Events) streaming support.

Container: claw-nginx (nginx:alpine) Host ports: 443 (HTTPS), 4000 → 80, and in prod also 80 → 80

The configuration is split across three files, all bind-mounted read-only:

File	Mounted at	Holds
infra/nginx/nginx.conf	/etc/nginx/nginx.conf	http{} settings, the port-80 block, the default (mkcert) TLS server block
infra/nginx/locations.conf	/etc/nginx/claw/locations.conf	every location — the routing table below
infra/nginx/public-tls/*.conf	/etc/nginx/claw/public-tls/	generated Let's Encrypt server blocks (gitignored; empty on a laptop)

The routing table is a separate file because a production server serves the same routes under two certificates — mkcert internally, Let's Encrypt for the public domain — and each TLS server block includes it. Duplicating ~52 location blocks per certificate guarantees the copies drift, and a drifted route is an outage that reproduces on only one hostname. See tls-setup.md.

---

Route Mappings

All routes are prefixed with /api/v1/. Nginx uses dynamic variables for upstream resolution to survive service restarts without stale DNS.

Auth Service (port 4001)

Frontend Path	Backend Destination	Notes
/api/v1/auth/*	http://auth-service:4001	Login, refresh, logout, me
/api/v1/users/*	http://auth-service:4001	User CRUD (admin only)

Chat Service (port 4002)

Frontend Path	Backend Destination	Notes
/api/v1/threads/*	http://chat-service:4002	Rewritten to /api/v1/chat-threads/*
/api/v1/messages/*	http://chat-service:4002	Rewritten to /api/v1/chat-messages/*
/api/v1/chat-threads/*	http://chat-service:4002	Direct path
/api/v1/chat-messages/stream	http://chat-service:4002	SSE endpoint (special config)
/api/v1/chat-messages/*	http://chat-service:4002	Message CRUD, feedback, regenerate

Important: The /api/v1/chat-messages/stream location must be defined BEFORE the general /api/v1/chat-messages location to ensure Nginx matches the more specific SSE route first.

Connector Service (port 4003)

Frontend Path	Backend Destination	Notes
/api/v1/connectors/*	http://connector-service:4003	Connector CRUD, test, sync
/api/v1/models/*	http://connector-service:4003	Model listing

Routing Service (port 4004)

Frontend Path	Backend Destination	Notes
/api/v1/routing/*	http://routing-service:4004	Policies, decisions, evaluate

Memory Service (port 4005)

Frontend Path	Backend Destination	Notes
/api/v1/memory/*	http://memory-service:4005	Rewritten to /api/v1/memories/*
/api/v1/context/*	http://memory-service:4005	Rewritten to /api/v1/context-packs/*
/api/v1/memories/*	http://memory-service:4005	Direct path
/api/v1/context-packs/*	http://memory-service:4005	Direct path

File Service (port 4006)

Frontend Path	Backend Destination	Notes
/api/v1/files/*	http://file-service:4006	Upload, list, chunks

Audit Service (port 4007)

Frontend Path	Backend Destination	Notes
/api/v1/audits/*	http://audit-service:4007	Audit log queries
/api/v1/usage/*	http://audit-service:4007	Usage statistics

Ollama Service (port 4008)

Frontend Path	Backend Destination	Notes
/api/v1/ollama/*	http://ollama-service:4008	Model management, pull, generate

Health Service (port 4009)

Frontend Path	Backend Destination	Notes
/api/v1/health	http://health-service:4009	Aggregated health

Grafana (port 3000, never published directly)

Frontend Path	Backend Destination	Notes
/grafana/*	http://grafana:3000	Behind auth_request → POST /api/v1/auth/grafana-access/verify (auth-service). See ADR-115.

Client Logs Service (port 4010)

Frontend Path	Backend Destination	Notes
/api/v1/client-logs/*	http://client-logs-service:4010	Frontend log ingestion

Server Logs Service (port 4011)

Frontend Path	Backend Destination	Notes
/api/v1/server-logs/*	http://server-logs-service:4011	Backend log viewer

Image Service (port 4012)

Frontend Path	Backend Destination	Notes
/api/v1/images	http://image-service:4012	Image generation requests
/api/v1/images/*	http://image-service:4012	SSE endpoint (special config)

Threads Service (port 4019)

Frontend Path	Backend Destination	Notes
/api/v1/thread-publications*	https://threads-service:4019	Public Thread publications; the legacy /api/v1/threads alias stays on chat

Plugin Service (port 4021)

Frontend Path	Backend Destination	Notes
/api/v1/plugins/*	https://plugin-service:4021	Catalog, installs, consent, credentials, admin. /api/v1/internal/* is never routed (closed by the 404 block)

---

SSE (Server-Sent Events) Configuration

Two endpoints use SSE for real-time streaming: chat message streaming and image generation progress.

Chat Message Streaming

location /api/v1/chat-messages/stream {
    set $chat_backend http://chat-service:4002;
    proxy_pass $chat_backend;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_read_timeout 86400;
    proxy_buffering off;
    proxy_cache off;
}

Image Generation SSE

location /api/v1/images/ {
    set $image_backend http://image-service:4012;
    proxy_pass $image_backend;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_read_timeout 86400;
    proxy_buffering off;
    proxy_cache off;
}

SSE Configuration Explained

Directive	Value	Purpose
proxy_http_version	1.1	Required for persistent connections (HTTP/1.0 closes after each response)
proxy_set_header Connection	""	Removes the Connection: close header to keep the connection open
proxy_read_timeout	86400	24-hour timeout to prevent Nginx from closing long-lived SSE connections
proxy_buffering	off	Disables response buffering so events are sent to the client immediately
proxy_cache	off	Disables caching of SSE responses

---

Timeout Settings

proxy_connect_timeout 10s;   # Time to establish connection to upstream
proxy_send_timeout    60s;   # Time to send request to upstream
proxy_read_timeout    60s;   # Time to read response from upstream (overridden for SSE)

These apply to all non-SSE routes. SSE endpoints override proxy_read_timeout to 86400s (24 hours).

---

Request Body Limits

client_max_body_size 75m;

This allows file uploads up to 75 MB. This limit applies to all routes. If you need larger uploads, increase this value and restart the nginx container.

---

Security Headers

Applied globally to all responses:

add_header X-Content-Type-Options "nosniff" always;
add_header X-Frame-Options "DENY" always;
add_header X-XSS-Protection "0" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
add_header Permissions-Policy "camera=(), microphone=(), geolocation=()" always;

Header	Value	Purpose
X-Content-Type-Options	nosniff	Prevents MIME type sniffing
X-Frame-Options	DENY	Prevents clickjacking (no iframe embedding)
X-XSS-Protection	0	Disables legacy XSS filter (modern CSP preferred)
Referrer-Policy	strict-origin-when-cross-origin	Controls referrer information sent with requests
Permissions-Policy	camera=(), microphone=(), geolocation=()	Disables access to sensitive browser APIs

Additionally, server_tokens off; hides the Nginx version from response headers.

---

Proxy Headers

Set on all proxied requests:

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Origin $http_origin;

Header	Value	Purpose
Host	$host	Preserves the original Host header
X-Real-IP	$remote_addr	Passes the client's real IP to backend
X-Forwarded-For	$proxy_add_x_forwarded_for	Appends client IP to the forwarded chain
X-Forwarded-Proto	$scheme	Indicates the original protocol (http/https)
Origin	$http_origin	Passes the Origin header for CORS handling

CF-IPCountry, CF-Connecting-IP and True-Client-IP are blanked next to them, so a client cannot pass one off as something the edge produced.

X-Real-IP is the throttler key — restate it in every block that sets headers

Every service's global throttler counts anonymous traffic by X-Real-IP (buildThrottlerOptions in @claw/shared-auth; rule: rules/08-security-rules.md §Rate limiting). nginx overwrites it with $remote_addr, so a client cannot choose its own key. X-Forwarded-For is never used as a key: nginx appends to it, and its left-most entry is client-controlled.

A location that declares any proxy_set_header (an SSE Connection "", a webhook Content-Type, the frontend Upgrade) inherits NONE of the server-level headers. Until 2026-10-02 the chat/agent/image/file-generation streams, the llama.cpp inference and pull-progress routes, the payment webhooks, the Grafana auth subrequest and the frontend catch-all all dropped X-Real-IP, so every visitor of those routes landed in one bucket. Each now restates the shared set; the frontend block restates only the client-address ones, to leave the Host the Next.js server sees unchanged. tools/__tests__/nginx-real-ip-restated.test.mjs fails on any block that sets headers without X-Real-IP $remote_addr, in both locations.conf and nginx.distributed.conf.template.

Verify live (forged headers must not move the counter, a second real client must start its own):

curl -sk -D - -o /dev/null -H 'X-Real-IP: 1.1.1.1' https://claw.local/api/v1/health | grep -i ratelimit-remaining
curl -sk -D - -o /dev/null -H 'X-Real-IP: 2.2.2.2' https://claw.local/api/v1/health | grep -i ratelimit-remaining   # one lower
MSYS_NO_PATHCONV=1 docker exec claw-nginx curl -sk -D - -o /dev/null -H 'Host: claw.local' https://127.0.0.1/api/v1/health | grep -i ratelimit-remaining  # fresh bucket

A call carrying the inter-service Service token gets no X-RateLimit-* headers: it skips the throttler.

---

Sign-in / sign-up flood brake (auth_routes, rules/58)

map $request_method $auth_post_key { default ""; POST $binary_remote_addr; }
limit_req_zone $auth_post_key zone=auth_routes:10m rate=60r/m;
# location /api/v1/auth  and  location /api/v1/agent/auth/
limit_req zone=auth_routes burst=30 nodelay;

• POST only. An empty key is not counted, so GET /auth/me on every page load never spends it.
• A backstop, not the limit. The real per-route budgets (login 10 / 15 min per IP + address, register 5 / h per IP, ...) live in auth-service and agent-service (rules/58, ADR-147). This zone is what still limits these routes when Redis is down and the services fail open.
• An nginx 429 has no code in its body and no service log line; a service 429 has code: RATE_LIMITED and Retry-After. Runbook: runbook-auth-rate-limits.md.
• Same zone, same location, in nginx.distributed.conf.template.

---

DNS Resolution

resolver 127.0.0.11 valid=5s ipv6=off;
resolver_timeout 3s;

• 127.0.0.11 is Docker's embedded DNS server
• valid=5s means DNS results are cached for 5 seconds, then re-resolved
• ipv6=off disables IPv6 resolution (not needed in the bridge network)
• resolver_timeout 3s sets the maximum time for a DNS query

This configuration allows Nginx to survive service restarts. When a service container is recreated, its IP address changes. The 5-second TTL ensures Nginx picks up the new IP quickly.

Why Variables Are Used

Each location block uses a variable for the upstream:

location /api/v1/auth {
    set $auth_backend http://auth-service:4001;
    proxy_pass $auth_backend;
}

This is required for dynamic DNS resolution. If the URL were passed directly to proxy_pass without a variable, Nginx would resolve the hostname at startup only and cache it indefinitely, causing failures when containers restart.

---

URL Rewriting

Some frontend paths differ from backend API paths. Nginx rewrites these:

Frontend Path	Rewritten To	Service
/api/v1/threads/*	/api/v1/chat-threads/*	chat-service
/api/v1/messages/*	/api/v1/chat-messages/*	chat-service
/api/v1/memory/*	/api/v1/memories/*	memory-service
/api/v1/context/*	/api/v1/context-packs/*	memory-service

The rewrite uses break flag, which means the rewritten URL is processed internally without an external redirect:

rewrite ^/api/v1/threads/(.*)$ /api/v1/chat-threads/$1 break;

---

Upstream Definitions

The current configuration does not use upstream blocks because each service runs as a single instance. If horizontal scaling is needed, add upstream blocks:

upstream chat-backend {
    server chat-service-1:4002;
    server chat-service-2:4002;
    server chat-service-3:4002;
}

Then replace proxy_pass $chat_backend; with proxy_pass http://chat-backend;.

---

Nginx Dependencies

Nginx waits for ALL backend services to be healthy before starting:

depends_on:
  auth-service: { condition: service_healthy }
  chat-service: { condition: service_healthy }
  connector-service: { condition: service_healthy }
  routing-service: { condition: service_healthy }
  memory-service: { condition: service_healthy }
  file-service: { condition: service_healthy }
  audit-service: { condition: service_healthy }
  ollama-service: { condition: service_healthy }
  client-logs-service: { condition: service_healthy }
  server-logs-service: { condition: service_healthy }
  health-service: { condition: service_healthy }

This means Nginx is the last infrastructure component to start, ensuring all routes are functional when the proxy becomes available.

---

Modifying the Configuration

1. Edit the right file:
   • adding or changing a route → infra/nginx/locations.conf. It is included by every TLS server block, so the route is served under both the mkcert and the Let's Encrypt certificate automatically. Never copy a location into a server block.
   • TLS, timeouts, rate-limit zones, security headers → infra/nginx/nginx.conf.
   • never hand-edit infra/nginx/public-tls/*.conf; it is generated by scripts/install-letsencrypt.sh and rewritten on every run.
2. Check the config parses before applying it — a bad reload on a running server is avoidable:

   docker exec claw-nginx nginx -t

3. Apply it. A reload is graceful and drops no connections:

   docker exec claw-nginx nginx -s reload

   A restart is only needed when the mount set itself changed (a new file or a new published port), and that requires a recreate, not a restart: ./scripts/claw.sh --prod up.
4. Verify routes work:

   curl -s -o /dev/null -w "%{http_code}\n" https://localhost/api/v1/health

All three files are mounted read-only (:ro), so edits on the host take effect on the next reload — but a file that was not mounted at container-create time will not appear by reloading.

After nginx -s reload, old workers keep accepting connections until they drain. A check fired immediately afterwards can still be answered by the old config; retry for a second or two before concluding a change did not apply.

X-Content-Type-Options (single header)

nginx.conf and nginx.distributed.conf.template run proxy_hide_header X-Content-Type-Options; before adding nosniff, so the copy services set themselves (helmet, download and stream controllers, kept for direct access) does not double up. A location with its own add_header stops inheriting the outer ones, so each such block re-adds nosniff. Guarded by tools/__tests__/nginx-nosniff-single.test.mjs.