Purpose of non-standard POST method in Websocket handshake?

[isaac-mcfadyen]

Hi!

First of all, thanks for all of the hard work that goes into Headscale. I use it daily for personal use and couldn't be happier. 🙏

I'm wondering about the use of the POST method in the Websocket handshake when connecting to Headscale. Per Section 4.1 of RFC 6455, "the method of the request MUST be GET". Further down the RFC specifies that invalid requests (such as methods other than GET) should be rejected.

I understand deviating from the spec if required - however, some proxies really don't handle Websocket handshakes via POST well (see this StackOverflow and this issue for lack of compatibility with Cloudflare Tunnel, a reverse proxy cloudflare/cloudflared#883).

I was wondering if there was a purpose making the Websocket handshake be a POST instead of the standard GET? 🙂

[juanfont]

Hi Isaac,

Unfortunately it is part of the tailscale control protocol.

[gbraad]

Is it related to the following?

https://github.com/tailscale/tailscale/blob/cb5384671729a02cd6eb9a8fa5a2e44af96c429b/control/controlhttp/client.go#L479-L487

Perhaps good to file an upstream issue for this.

---

@kradalby WDYT?

[kradalby]

Our general take on this is that we do not support proxies, and this is one of the reason.

In the end it is more of a custom protocol than a RFC protocol, and while an issue could be raised upstream, it would probably not be a change that there is intensive to fix as it would break (or make more surface to maintain) backwards compatibility.

The social contract between Headscale and Tailscale is based on not causing Tailscale additional work and while they very much welcome changes that improve stuff for both users, we do not really want to push these boundaries to maintain our good relationship.

If issues are raised upstream, my take is that they should not solely improve life for Headscale, and they should not cause additional work for Tailscale.

[kradalby]

I'm going to close this as out of scope.

[lukeIam]

idea for workaround: can we identify the requests with missing headers and "repair" them with an additional reverse proxy?
I added a nginx reverse proxy between my headscale and cloudflared cointainers.
But adding the headers to all POST requests to /ts2021 seems to be too much and causes other errors...
@isaac-mcfadyen @gbraad

[Ducky6944]

If issues are raised upstream, my take is that they should not solely improve life for Headscale, and they should not cause additional work for Tailscale.

Just for POSTerity, the usage of POST here is antithetical to the Websockets RFC.

https://www.rfc-editor.org/rfc/rfc6455#section-4.1

The method of the request MUST be GET, and the HTTP version MUST be at least 1.1.

I, very respectfully, believe that this improves tailscale and is largely inconsequential for most everyone else.

Cloudflare doesn't want to break from RFCs, and Headscale doesn't want to burden Tailscale. I suspect this affects a large number of newcomers to Headscale, who probably hit this point are unable to effectively troubleshoot it, or see the situation as described and just abandon their attempts rather than raising an issue. Though it doesn't seem widely reported, Cloudflare is ubiquitous for homelabbers, small businesses, etc, particularly in the US. It's reasonable for headscale to not want to "support" proxies, but also reasonable that tailscale adheres to RFCs unless theres a specific reason not to.

I apologize for bumping a closed thread. I'm not trying to get into a conversation, disrespect, or point fingers. Regardless of workarounds, raising issues with cloudflare/tailscale etc, it might be nice to put a disclaimer high up on the repos' readme for visibility. I do see the reverse proxies notification, but also many proxies, guides on the internet, discussions on reddit etc use them. Maybe linking to the issue or something would be valuable to some. Just a thought.

[alx-xlx]

That is why people always use an opensource project or repo, that sticks to open standards.
this repo is no longer opensource

Opensource has 4 requirements:

1. use the code
2. study the code
3. modify the code
4. share the code

This repo doesn't tick the 3rd box.
Yes, You can modify the code however it will be unusable with the closed tailscale clients. so pretty much you can't modify the code.

This is how they get you !!

I bet many people chose tailscale seeing the headscale as alternative to their control server. Hoping to maybe one day spin their cloudflared tunnel to run headscale. Gotcha !!!

strategically headscale is in tailscale pockets.

[juanfont]

strategically headscale is in tailscale pockets.

The purpose of this project is to allow people to re-use the Tailscale client with a control server of their own. For that, we need to reimplement the Tailscale control protocol. If we don't do that correctly, the Tailscale clients won't work. If that's being in "Tailscale pockets" so be it.

Tailscale uses the standard websocket library from Go to hijack the TCP connection and create a Noise session over it between server and client. That does not mean that "Tailscale uses websockets".

We mention Websockets because the config in reverse proxies that enable websockets to work usually will allow Headscale/Tailscale to work. But that's not granted. And we even mention the Cloudflare case:

https://headscale.net/stable/ref/integration/reverse-proxy/#cloudflare

[luckycreationsindia]

Hi,

I don't know why it didn't work for some people out there. It's working great for me even with cloudflare DNS proxy.

I'm using nginx for reverse proxy like below,

http://localhost:8080 -> headscale

location / {
        proxy_pass http://localhost:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        proxy_buffer_size 128k;
        proxy_buffers 4 256k;
        proxy_busy_buffers_size 256k;
    }

It's working great for me without issues. I do have proxy enabled in cloudflare DNS.

[jimstrang]

@luckycreationsindia - you are right there's no issues with the DNS proxy but people are having issues with the full cloudflare tunnel (cloudflared) proxy

[luckycreationsindia]

@luckycreationsindia - you are right there's no issues with the DNS proxy but people are having issues with the full cloudflare tunnel (cloudflared) proxy

So it's Cloudflared Tunnel issue and not Cloudflare DNS Proxy.

Maybe we need to be more clear about it in docs.

[dreamstarsky]

Using POST to open a WebSocket works well on Alibaba Cloud ESA (though there are still issues with long connections disconnecting).