Recipe Reverse Proxy

Recipe: reverse proxy

For Rails/Django/Express/Next.js/anything-that-renders-HTML behind Caddy. The static-first path doesn't apply — there's no root on disk — so the plugin runs in capture-and-convert mode: it wraps the response, captures the HTML, converts it, caches the result in memory (LRU only, no disk sidecar), and serves the Markdown.

The shape

agent ──Accept: text/markdown──> Caddy ──HTML──> backend
                                   │
                                   └── convert + cache + serve as MD

The backend is unaware that the request is for Markdown. It always receives an HTML request (the URL suffix and query param are stripped before reverse_proxy sees them). Same URL, same auth, same routing.

Caddyfile

example.com {
    markdown_for_agents {
        main_selector main
        strip_selectors nav footer ".site-chrome"
    }
    reverse_proxy backend:8080
}

That's it. No root. The module sees no root, infers dynamic mode, and wraps reverse_proxy.

What gets cached, what doesn't

The dynamic path applies real HTTP caching semantics. A response is not cached if any of the following:

• Request was not GET or HEAD
• Request carried Authorization or Cookie (unless you set allow_authenticated)
• Upstream sent Set-Cookie
• Upstream sent Cache-Control: private or no-store
• Upstream sent a Vary value other than Accept-Encoding
• Response body exceeded max_body_bytes (default 4 MiB)
• Response was an oversized entry (exceeded cache_entry_bytes, default 1 MiB)

In all those cases the conversion still happens — the agent still gets Markdown — but the result is served once and discarded.

Cache key

For the dynamic path the key is path + ?query. So:

/api/post?id=1
/api/post?id=2

…are independent entries. Query order isn't normalized, so prefer canonical query strings if your app permits.

When upstream returns non-HTML

Status code is 204, 3xx, or 5xx? Pass through unchanged.

Content-Type is anything except text/html? Pass through unchanged. Image responses, JSON APIs, RSS feeds — none of them get touched.

Personalized content

The Authorization and Cookie checks catch the obvious cases. If your app personalizes on other signals — mTLS client cert, X-Forwarded-User from upstream auth, IP-based ACL — those are not automatically considered cache-unsafe. Two options:

1. Have the upstream set Cache-Control: private on personalized responses. The module honors it.
2. Run separate module instances per personalization dimension (e.g. one route per tenant).

Trimming chrome from a dynamic backend

The strip_selectors directive runs the same way as in static mode. Concrete examples:

Rails app with the default Sprockets nav

markdown_for_agents {
    main_selector "#content"
    strip_selectors ".navbar" ".flash" ".footer-links"
}
reverse_proxy rails:3000

Next.js with a header/footer layout

markdown_for_agents {
    main_selector main
    strip_selectors "header" "footer" ".cookie-consent"
}
reverse_proxy next:3000

WordPress / WooCommerce

markdown_for_agents {
    main_selector ".entry-content"
    strip_selectors \
        ".wp-block-buttons" \
        ".sharedaddy" \
        "#comments" \
        ".related-posts" \
        nav \
        footer
}
reverse_proxy wp:80

Public + authenticated mix

If most of your routes are public but a few require login, the safe default (cache-bypass on Cookie/Authorization) does the right thing automatically — public agent traffic gets cached, authenticated traffic doesn't.

If you've got a full SaaS where every request is authenticated and content is still identical for all users on a given URL (rare, but e.g. landing pages behind a marketing-site CDN gate), you can opt in:

markdown_for_agents {
    allow_authenticated
}
reverse_proxy backend:8080

Read Configuration carefully before flipping this. The wrong call here serves User A's session-rendered page to User B's agent.

Conversion timeout & concurrency

For backends with long tail-latency responses, bump the timeout:

markdown_for_agents {
    convert_timeout 15s
    max_concurrent  32
}

max_concurrent is a semaphore over conversions, not over upstream requests. The upstream still sees its normal request rate; this only limits how many HTML→Markdown passes run in parallel. Default is max(4, NumCPU).

Test it

Start a fake backend for a smoke test:

docker run --rm -d --name fake -p 8080:80 nginxdemos/hello

Minimal Caddyfile pointing at it:

:80 {
    markdown_for_agents
    reverse_proxy localhost:8080
}

Then:

curl -sI http://localhost/ -H "Accept: text/markdown"
# Content-Type: text/markdown; charset=utf-8
# Vary: Accept
# ETag: "..."

curl http://localhost/ -H "Accept: text/markdown"
# (the nginxdemos hello page, rendered as Markdown)

When not to use reverse-proxy mode

If your backend is just serving static files (e.g. a Hugo build), prefer the static-site recipe. The static-first path gives you:

• Persistent disk sidecars (capture-and-convert is in-memory only)
• Author-written .md precedence
• mtime-based invalidation
• Lower per-request overhead

See also

• Recipe: static site
• Configuration
• FAQ