Troubleshooting

Troubleshooting

Diagnostics

• Effective configuration: mitm-proxy config prints every setting and where it came from (default, file, env var). Secrets are masked.
• More logging: MITM_PROXY_LOG_LEVEL=DEBUG.
• Health: curl http://127.0.0.1:8080/healthz (alive) and /readyz (ready to serve).
• Container: docker logs mitm-proxy, and docker inspect --format '{{json .State.Health}}' mitm-proxy.
• Kubernetes: kubectl -n mitm-proxy logs deploy/mitm-proxy and kubectl -n mitm-proxy describe pod.
• systemd: journalctl -u mitm-proxy -e and systemctl status mitm-proxy.

Common problems

The client says the certificate isn't trusted

The client doesn't trust the proxy's root CA. Install it: download http://127.0.0.1:8080/ca.pem (or /ca.der), or run mitm-proxy ca show. The proxy logs TLS handshake with … failed …: if the client rejected the certificate, install the root CA on it. Some apps pin certificates and will never accept it: remove their hosts from intercept.hosts.

A host isn't intercepted (no flow, no middleware)

Run mitm-proxy scope check HOST. Remember *.example.org doesn't match example.org itself. The client's TLS name (SNI) must be listed too: mitm_proxy_tunnels_total{reason="sni_out_of_scope"} counts these. Apps that ignore the system proxy settings, or use HTTP/3, never reach the proxy at all.

502 … certificate verification failed

The origin's certificate doesn't verify: it's self-signed or from a private CA, or doesn't name the host. Add the CA to upstream.extra_ca_file. Verification can't be switched off.

Every request gets 407 Proxy Authentication Required

proxy.auth is set and the client sends no or wrong credentials. Use http://user:password@host:3128 as the proxy URL, or curl --proxy-user.

An origin answers 411 Length Required

A middleware that rewrites bodies applies to the flow, so the request body is sent chunked, and this origin refuses chunked uploads. Narrow the middleware with applies_to so it skips these requests.

middleware 'x' failed in … / timed out / returned …, not … in the log

The middleware raised, ran past middleware.hook_timeout_ms (or its own timeout), or returned the wrong type (the log names the type, never the value). The flow carried on without it, unless the middleware sets on_error = "fail": then the flow ended with 502 middleware x failed (outcome denied:x), as a gate should. Fix the middleware; the traceback is in the log. The timeout is per hook call (each chunk-hook call too), not per flow, and it cancels the hook at its current await. A middleware that blocks the event loop (sync I/O, time.sleep, heavy CPU) can't be timed out and stalls everything: move that work to asyncio.to_thread (the thread keeps running after a timeout). A timeout line that ends late: overran means the hook ran synchronous code past its deadline; late: swallowed means it caught the cancellation and returned anyway. Either way its result was discarded and it's treated as a timeout.

middleware 'x' returned 'bytes' from on_response_chunk, but it is observe-only in the log

The middleware sets observe_only = True but its chunk hook returned bytes, as if to change the body. Observe-only hooks must return None; the traffic carried on untouched and the middleware was skipped for the rest of that flow. Return None, or drop observe_only if it really rewrites bodies (its flows then buffer). See Writing Middleware.

observe_only can't be combined with on_error = 'fail' at startup

A middleware sets both. On a streaming flow its chunk hooks run after the response head has gone, so a failure can't become a 502. Keep on_error at its default for an observe-only middleware; a gate that must fail closed should not be observe-only.

A middleware gets RuntimeError: a streamed body is not buffered

It called read_all() on a flow where every middleware is observe-only, so nothing holds the body. Read the body from on_request_chunk / on_response_chunk instead, or, if it must have the whole body in one piece, don't mark it observe-only (its flows then buffer).

middleware 'x': on_request must be an async function at startup

One of the middleware's hooks is a plain def. Every hook except applies_to must be async def (design 0011). The proxy refuses to start; a new configuration with it is refused and the running one stays.

Requests are slow: the proxy, the origin or a middleware?

Turn on the access log (proxy.access_log: true) and read the end of each flow line:

… 200 req=0 resp=1256 total=812.4ms connect=153.0ms ttfb=640.2ms mw=0.4ms outcome=origin

• ttfb is the origin: from the request being sent to its response headers. If it's most of total, the origin is slow, not the proxy.
• connect is waiting for an upstream connection. Without (reused), it's a new TCP and TLS connection, and high values point at DNS or the network to the origin. With (reused) it's normally about 0; a high value means the request queued for a free connection because the host had upstream.max_connections_per_host busy ones.
• mw is time spent in middleware hooks. To find which one, read flow.timings.middleware (see Writing Middleware). On a flow that streams past observe-only middleware (such as capture alone), the chunk hooks run while the body moves, so their time is also inside the remainder below.
• The rest of total is the proxy's own work plus moving the bodies: receiving the response from the origin and sending it to the client. A large remainder on a big response is the download, not the proxy.

- means that stage didn't happen (for example, connect=- on a response a middleware sent itself, outcome=local:<name>).

For the trend rather than one request, the same stages are histograms on /metrics: mitm_proxy_origin_ttfb_seconds going up means the origin got slower; mitm_proxy_upstream_connect_seconds{reused="false"} going up points at the network or the origin's TLS; mitm_proxy_middleware_duration_seconds by middleware finds a slow one. Example queries are in HTTP API Reference, under GET /metrics.

In a browser, set proxy.server_timing: true and open devtools (Network → a request → Timing): the same breakdown appears under Server Timing as mitm-connect, mitm-origin, mitm-mw and mitm-proxy, and mitm-local names the middleware that answered a request itself. These show up even for responses no middleware ran on. On a streamed flow (every middleware observe-only) they're written with the head, before the body: mitm-proxy is the time to the head, not the whole response, and mitm-mw leaves out the response's chunk hooks. See In-Depth Configuration.

504 the origin did not answer in time from a slow origin

The origin sent nothing for upstream.read_timeout_ms (default two minutes): no response headers, or a pause that long in its body. A WebSocket upgrade gets the same 504 when the origin doesn't answer the upgrade in that time; once it has switched protocols, the relay is idle traffic and stays on proxy.idle_timeout_ms. Some origins are slow by nature, LLM APIs above all, which can think for minutes before the first byte. Raise upstream.read_timeout_ms (300000 for five minutes); a reload applies it to the requests that start afterwards. Don't raise proxy.idle_timeout_ms for this: it doesn't end an open request anyway, and a larger value only keeps idle client connections and tunnels open longer. A 504 with connecting to … timed out or no free connection is upstream.connect_timeout_ms instead. See In-Depth Configuration, under Timeouts.

A flow ends with outcome=aborted

The flow ended before the whole response was sent. Usually the client gave up: a browser cancels image and media requests it no longer needs (an HTTP/2 stream reset), or the user navigated away. Occasional aborted lines are normal. If every flow to one host ends that way, check the proxy's warnings for that flow ID: an origin that stops sending part-way also ends here.

… body is over body.max_buffer_bytes …; streaming it through without middleware

That body was bigger than body.max_buffer_bytes, so middleware didn't see it. Raise the limit if you need to inspect such bodies (they're held in memory). Only buffered flows have the limit: if every middleware on the flow only reads bodies, mark them observe_only and the flow streams with no limit (Writing Middleware).

capture: dir … can't be created or written / a record for rule … can't be written

The capture middleware can't write to its dir. The first line is at startup: the proxy runs anyway. The second is for a record it dropped, logged at most once a minute per rule; mitm_proxy_capture_records_total{result="failed"} counts every one. The traffic is never affected. Check that dir exists or can be created, that the proxy's user can write it (in a container, that the volume is mounted read-write; under systemd, that it's under /var/lib/mitm-proxy or in ReadWritePaths=), and that the disk isn't full. A dropped record isn't written later: the flows in between are lost.

Capture recorded a value it should have redacted

json_paths apply only to bodies declared application/json or *+json that parse as JSON, and to the data: lines of a text/event-stream body whose value parses as JSON; other bodies and lines are stored as they are, silently. Headers are redacted only by name: a list you give replaces the default five. Add the header name, or narrow the rule (path, methods) to the JSON traffic, and test with a fake credential (see Writing Middleware).

the proxy listens on 0.0.0.0 without proxy.auth

Anyone who can reach the port can use the proxy. Set proxy.auth, or bind 127.0.0.1.

error: cannot listen on 127.0.0.1:3128: Address already in use

Another process has the port (ss -ltnp 'sport = :3128'). Change proxy.port or listen.port.

Clients must re-trust the CA after every restart

The CA is generated on first start in <state_dir>/ca/, and state_dir isn't persistent (a container without a volume, a pod on emptyDir). Persist it, or supply the CA with ca.cert/ca.key.

Responses from the proxy

Responses the proxy sends itself, with a one-line plain-text body:

Status	Meaning
400	Malformed request, a relative URL sent to the proxy, or CONNECT without host:port
403	denied by middleware <name>: a middleware's deliberate deny on the request
405	CONNECT where it isn't allowed
407	Proxy credentials missing or wrong
501	CONNECT over HTTP/2 (not supported)
502	The origin couldn't be reached, failed verification, or broke the connection; denied by middleware <name> (a deny on the response); middleware <name> failed (a fail-closed middleware failed, in either phase); a middleware stopped the request without a response
504	The origin didn't connect or answer in time

error: <key>: expected int (or str, bool, list)

A value doesn't have its setting's type, and the message says where it came from (file, or the variable's name). The usual causes: a decimal where a whole number is wanted (port: 3128.0, max_buffer_bytes: 0.5), digits with underscores in a variable (1_000), or a YAML list where one value is wanted (listen.host: [a, b]). Versions before design 0008 silently truncated a decimal, read 1_000 as 1000 and used a list's text as the value; now all three stop startup. Fix the value: mitm-proxy config shows every setting's source. For a secret setting, the message leaves out the value.

error: cannot read MITM_PROXY_..._FILE

The file named by a _FILE variable doesn't exist, or the process can't read it. Check the path inside the container or unit, and the file's owner and mode: the process runs as uid 1000 / the mitm-proxy user.