Immich behind HTTP/2 reverse proxy supporting extended CONNECT fails to connect WebSocket in web UI at least in Firefox

[Domini]

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

• Yes

The bug

There are two ways of WebSocket connection

• HTTP/1.1 via Upgrade: websocket
• extended CONNECT for HTTP/2

Upgrade: websocket is explicitly disallowed for HTTP/2

Note also that HTTP/2 explicitly disallows the use of this mechanism; it is specific to HTTP/1.1.

One can check if their reverse proxy supports extended CONNECT for HTTP/2 using e.g. nghttp -nv https://hostname | grep -F SETTINGS_ENABLE_CONNECT_PROTOCOL

[SETTINGS_ENABLE_CONNECT_PROTOCOL(0x08):0] # not supported
[SETTINGS_ENABLE_CONNECT_PROTOCOL(0x08):1] # supported

Different browsers work with reverse proxies supporting extended CONNECT for HTTP/2 differently. I haven't checked all of them, but so far the newest version of

• Chrome instantiates WebSocket connection using HTTP/1.1 via Upgrade: websocket even if the server supports extended CONNECT and even if other, non-WebSocket connections to this server use HTTP/2 or HTTP/3 (I assume because Chrome has not yet implemented the support for extended CONNECT, but I might be wrong)
• Firefox has an about:config setting network.http.http2.websockets - if disabled it leads to the same behaviour as Chrome
• yet by default this setting is enabled in Firefox and if the server does not support extended CONNECT - the same behaviour as Chrome
• but if the server does support extended CONNECT - Firefox uses HTTP/2 for WebSocket connection but instead of using CONNECT method it sends Upgrade: websocket header which is explicitly against the RFC 8441.

On one hand this kind of happens because of a race condition - the request data (including headers) can be sent to HTTP/2 connection before you get its SETTINGS frame, so Firefox kind of does not know whether the server supports extended CONNECT before it sends Upgrade: websocket. On the other hand, when server supports HTTP/2 but does not support extended CONNECT, Firefox somehow understands it needs to use HTTP/1.1 for WebSocket, so for some reason race condition no longer leads to an error. Also when it connects using HTTP/2, it knows that Upgrade: websocket is against the standard in any case, but still sends it. So there is wrong behaviour for sure, the question is how it should be fixed. And no matter what, after failing once it should understand that the next time it should either reconnect via HTTP/1.1 if it wants to continue using Upgrade: websocket, or use extended CONNECT if it wants to continue using HTTP/2 - there is no race condition on the following attempt.

As far as my investigation went, Immich uses socket.io, which uses browser native websockets/ws implementation, though I might be wrong. I'm not sure which layer should be responsible for fixing the error. As far as I understand the linked chain above, the error happens because the WebSocket implementation is not at all aware that HTTP/2 requires different WebSocket method as none of the layers have implemented HTTP/2 extended CONNECT yet.

Here comes the fuzzy part of the issue. My Immich is behind envoy reverse proxy which is very strict in following the standards - it responds with 400 Bad Request without even proxying the request to Immich. All the WebSocket connections from Immich fail. (For anyone encountering the same issue, temporary workaround is to disable extended CONNECT (http2_protocol_options.allow_connect=false) at least for the /api/socket.io/ prefix - Firefox will start using HTTP/1.1 for Immich WebSocket connections and continue using HTTP/2 for the rest.)

I understand that envoy is not a supported configuration for Immich (there is no example for envoy in the docs), but setting up another reverse proxy just to check whether the behaviour is the same is too much work.

Also, as far as I understand, nginx has just recently implemented extended CONNECT (not yet merged), so the problem might appear soon out of nowhere as the fleet of nginxs is auto-updated even if there is no problem now. As far as I understand, at the moment nginx will simply respond with SETTINGS_ENABLE_CONNECT_PROTOCOL=0 and everything will work - not because browser works better with nginx, but rather because nginx does not support the technology that leads to the wrong behaviour.

I don't know about Apache and Caddy.

If you have an nginx, Apache, Caddy or other reverse proxy HTTP/2-enabled configuration, could you check what nghttp -nv https://hostname | grep -F SETTINGS_ENABLE_CONNECT_PROTOCOL says, and if it's SETTINGS_ENABLE_CONNECT_PROTOCOL=1 whether Immich web UI WebSockets work in Firefox.

The OS that Immich Server is running on

Fedora CoreOS stable latest (based on Fedora 44 so pretty bleeding edge)
Firefox 154.0 on macOS Sequoia 15.7.9

Version of Immich Server

v3.1.0

Version of Immich Mobile App

Platform with the issue

• Server
• Web
• Mobile

Device make and model

No response

Your docker-compose.yml content

-

Your .env content

-

Reproduction steps

1. setup Immich behind reverse proxy supporting HTTP/2 with extended CONNECT
2. open Firefox dev console
3. open Immich web UI
4. log in
5. check if WebSocket connection works

Relevant log output

Additional information

This might not be immediately fixable issue, but I think it's worth having it open to track the issue if encountered by others or in different configurations.

[mmomjian]

Moving to FR for now, as mentioned it’s not clear where/if there is an actual bug here and this is more of a data gathering stage. For now I would say Immich does not support this newer websocket setup.

[bo0tzz]

It seems to me like the interaction/problem here is entirely between the browser and reverse proxy, and Immich isn't really involved in any way that we could do something about?

[Domini]

Not exactly, Immich web UI is triggering the WebSocket connection, so something could be done. However, I agree with you if the question is rather "is Immich the right layer to fix this?".

[bo0tzz]

Yes Immich uses websockets, of course, that doesn't really change much about what I said though - what do you imagine could be done...?

[Domini]

E.g. check reason for error, try forcing HTTP/1.1 on reconnect. That being said, it seems to be dirty workaround at the wrong layer.

[bo0tzz]

That's not just a dirty workaround, it's impossible for us to do at all.

[Domini]

Yes, as I see even low-level native browser WebSocket API behind socket.io does not allow this.

It should be possible to replace native browser implementation with a dependency that implements WebSocket better, but that's huge overkill for this fix.