Support the use of a reverse proxy with CUPS

[siepkes]

I would like to put a reverse proxy in front of CUPS to deal with "misbehaving" clients. For example clients which are slow senders (or receivers) and thereby blocking CUPS for all other clients. With NGINX you can for example use proxy_request_buffering on; and proxy_buffering on; to buffer the entire request (and response) body before sending it to CUPS (or back to the client). This should shield CUPS from clients which are slow senders and/or slow receivers. A reverse proxy can also help with things like request rate limiting. Some Linux clients sometimes start sending dozens of request per second ( https://gitlab.gnome.org/GNOME/xdg-desktop-portal-gnome/-/issues/201 ). A reverse proxy can also do the TLS termination.

However I can't get a working reverse proxy setup with CUPS. I think this is mainly because CUPS does not support XFF headers such as X-Forwarded-For, X-Forwarded-Host, X-Forwarded-Proto, etc. From what I could gather this results in CUPS "seeing" the reverse proxy as the client and generating URL's like http://127.0.0.1:632/printers/pr0001 in the response. I can get as far as seeing all the printers shared by CUPS, but as soon as I try printing with them in applications like Gnome (for example the PDF viewer) or Chrome things fall apart.

I tried things in NGINX like sub_filter 'http://127.0.0.1:632' 'https://print.nl2.serviceplanet.nl:631'; to rewrite these URL's in the IPP responses, but ultimately none of this resulted in a working solution.

All information online about using a proxy server with CUPS is (as far as I can tell) just about the admin web interface. Not about doing actual IPP printing via the reverse proxy.

This is a snippet from NGINX config I tried (one of many), to give some idea of what I was trying to accomplish:

server {
    listen       631 ssl;
    server_name  print.nl2.serviceplanet.nl;
    
    # Contains 'ssl_protocols' config and such.
    include tls-options.conf;
    # SSL / TLS certificate config for the HTTPS listener.
    ssl_certificate     /usr/local/etc/nginx/ssl/print-nl2.crt;
    ssl_certificate_key /usr/local/etc/nginx/ssl/print-nl2.key;
        
    location / {
        # Buffer the full request before sending it to CUPS. This shields CUPS
        # from clients which are slow in sending their requests. A single slow
        # sender or receiver can make CUPS slow for all clients.
        proxy_request_buffering on;
        # Also enable response buffering. If the client is slow to read the
        # response this could also hinder CUPS.
        proxy_buffering on;
        # Buffer requests up to 15MB in memory. Rest goes to a temporary file.
        client_body_buffer_size 15m;
        client_max_body_size 128m;
            
        proxy_pass http://127.0.0.1:632;
            
        proxy_http_version 1.1;
        proxy_set_header Connection "";  
    }        
}

[michaelrsweet]

Sorry, it is unlikely in the extreme that we will ever support reverse proxies with CUPS or IPP services in general. Some specific issues:

1. The "X-" headers are non-standard and would require explicit addition to the CUPS API since libcups' HTTP API throws away unknown headers.
2. The scheduler (cupsd) would need explicit support for reverse, not only for the new headers but for specifying allowed reverse proxy hosts, plus any additional changes to the existing DNS rebinding attack mitigations.
3. Any reverse proxy needs to handling chunked requests, caching/relaying gigabytes of request/response data, and different kinds of HTTP authentication/authorization.
4. Proxying of this kind will have a noticeable affect on print performance (read: users will complain).

Finally, I am not aware of any IPP deployment successfully making use of HTTP proxies, and CUPS has never supported the use of traditional (forward) HTTP proxies nor SSH port forwarding or other mechanisms.

[siepkes]

Fair enough, though I don't really understand the reasoning.

Proxying of this kind will have a noticeable affect on print performance (read: users will complain).

As far as I can tell from looking at the source and from what I've read, currently due to the way CUPS is designed, CUPS can't reliably deal with misbehaving clients (just misbehaving, not even malicious). For example if there are a couple of clients which send their requests slowly, CUPS will hang and no clients will get any response. I am currently unable to create a stable CUPS server setup (i.e. one that doesn't hang a lot) with even 40 Gnome desktop clients. Simply because there is often some client misbehaving. For example due to wifi conditions, bad behavior of the Gnome flatpak sandbox integration, etc. So I already have complaining users.

Any reverse proxy needs to handling chunked requests, caching/relaying gigabytes of request/response data,

That is the thing which reverse proxies like NGINX, Envoy, HAProxy, etc. do. Pretty much al HTTP servers have reverse proxies in front of them in typical deployments. Ranging from small / medium deployments (Proton mail, ibm.com) to large deloyments (github.com, amazon.com, etc.). They typically support just about every aspect of the HTTP protocol you can think of.

and different kinds of HTTP authentication/authorization.

Aside from mTLS (since it must be done when the TLS connection is terminated at the proxy) a HTTP server behind a reverse proxy can do all the forms of authentication it wants. A reverse proxy like NGINX, Envoy, etc. will just pass it along.

not only for the new headers but for specifying allowed reverse proxy hosts,

Usually applications which often sit behind reverse proxies (Apache HTTPD, Jetty, Apache Tomcat, Envoy, etc.) allow configuring if the application honors XFF headers or not. If an application sits behind a reverse proxy it is normally only available through the reverse proxy, since the reverse proxy is intended to be the "castle" around the application.

plus any additional changes to the existing DNS rebinding attack mitigations.

XFF headers are either trusted or not. By default they are usually not trusted (and ignored) unless the configuration says they need to be honored. If they are honored it is a reasonable assumption the application sits behind a trusted reverse proxy and the XFF headers are to be trusted.

The "X-" headers are non-standard

They are defacto standards supported by virtually all webservers and reverse proxies. But there is also the RFC 7239 Forward header as an alternative, which does the same thing and is a standard.

[michaelrsweet]

As far as I can tell from looking at the source and from what I've read, currently due to the way CUPS is designed, CUPS can't reliably deal with misbehaving clients (just misbehaving, not even malicious). For example if there are a couple of clients which send their requests slowly, CUPS will hang and no clients will get any response. I am currently unable to create a stable CUPS server setup (i.e. one that doesn't hang a lot) with even 40 Gnome desktop clients. Simply because there is often some client misbehaving. For example due to wifi conditions, bad behavior of the Gnome flatpak sandbox integration, etc. So I already have complaining users.

Current cupsd employs non-blocking socket connections and short (1-10 second) timeouts for things like TLS records and IPP attributes, with improvements made in very recent releases. The maximum number of simultaneous clients is limited to one third of the maximum number of file descriptors allowed per process (ulimit -n and then divide by 3), with a hard TCP limit of a little over 60k clients (source port is 16 bits).

Adding a reverse proxy just masks the issues and will delay all valid requests by a measurable amount of time (store and forward in both directions).

That is the thing which reverse proxies like NGINX, Envoy, HAProxy, etc. do. Pretty much al HTTP servers have reverse proxies in front of them in typical deployments. Ranging from small / medium deployments (Proton mail, ibm.com) to large deloyments (github.com, amazon.com, etc.). They typically support just about every aspect of the HTTP protocol you can think of.

Web traffic is different from IPP traffic.

Aside from mTLS (since it must be done when the TLS connection is terminated at the proxy) a HTTP server behind a reverse proxy can do all the forms of authentication it wants. A reverse proxy like NGINX, Envoy, etc. will just pass it along.

cupsd supports a lot of local (domain socket) methods, and AFAIK Kerberos won't work with a reverse proxy.

Usually applications which often sit behind reverse proxies (Apache HTTPD, Jetty, Apache Tomcat, Envoy, etc.) allow configuring if the application honors XFF headers or not. If an application sits behind a reverse proxy it is normally only available through the reverse proxy, since the reverse proxy is intended to be the "castle" around the application.

I get that, but right now that configurability does not exist and would need to go through a lot of review since it is effectively a "back door" through any of the existing security features (such as DNS rebinding mitigations).

They are defacto standards supported by virtually all webservers and reverse proxies. But there is also the RFC 7239 Forward header as an alternative, which does the same thing and is a standard.

"De-facto" is not the same as a standard that has widespread security review, acceptance, and interoperability.