FR: Support stripping HTTP request paths from funnel proxy routes

[mvdan]

What are you trying to do?

I'm using Funnel to replace ngrok with tailscale. So far it's been working well!

Right now, I have set it up as follows:

# tailscale serve status
https://p14s.blue-toad.ts.net (Funnel on)
|-- / proxy http://127.0.0.1:10080

Right now I am using this to test HTTP webhooks from a GitHub App, where the local Go server simply listens on http at port 10080. GitHub sends the events to https://p14s.blue-toad.ts.net/. So far so good, and it works perfectly.

However, this particular GitHub app is only one of the pieces of software that I want to expose via Funnel for end-to-end testing. I could swap which program listens on port 10080 each time, but that's not great. For example, if I stop my Go backend for that GitHub App and work on something else serving HTTP over funnel, it will still receive the GitHub App webhook events from the previous app, because they're still wired up to the same place.

I could disable the webhook events from the GitHub App every time I stop testing it, but that's cumbersome. The problem here is that I'm sharing the same host (https://p14s.blue-toad.ts.net/) to test multiple apps at different times.

One decent option is to use different ports:

  --serve-port uint
    	port to serve on (443, 8443 or 10000) (default 443)

However, that's only three ports, so I could only test three different pieces of software, and then we're back to the same problem of sharing the same host and port. It would also be confusing for future me: I'll be sure to forget what I used each port for, even if I had unlimited ports. Potentially, attaching some form of label or name to each route or port, but that wouldn't be visible when looking at the HTTPS URLs alone.

I tried to solve this by using different paths, like so:

# tailscale serve status
https://p14s.blue-toad.ts.net (Funnel on)
|-- /                     text  "p14s is online"
|-- /cue-unity-github-app proxy http://127.0.0.1:10080

This way, the GitHub App was configured with the URL https://p14s.blue-toad.ts.net/cue-unity-github-app, which is not going to be shared with any other funnel route. The path also helps me remember what I exposed this particular route for. However, when doing a GET on that URL, an HTTP server listening on 10080 sees:

$ nc -l 127.0.0.1 10080
GET /cue-unity-github-app HTTP/1.1
Host: p14s.blue-toad.ts.net
[...]
X-Forwarded-For: 100.93.97.75

That is, it receives a GET /cue-unity-github-app rather than GET /. Right now it expects webhooks to come in at POST /foo, not at POST /cue-unity-github-app/foo. I could teach each one of these pieces of software to strip a path prefix, like https://pkg.go.dev/net/http#StripPrefix, but it feels unfortunate. Some of the pieces of software I need to test aren't written in Go, so changing their behavior won't be as straightforward.

How should we solve this?

Funnel already acts as an HTTP proxy, so I think it could also be taught to strip paths so that my app would see GET / in the example above. It doesn't need to be always on, or even by default - it could be an opt-in flag, like --strip-prefix.

Note that this is a problem I experience with the proxy subcommand, but not with path. If /home/mvdan/foo/bar.txt exists as a plaintext file and I expose it via tailscale serve /foo path /home/mvdan/foo, you will notice that https://p14s.blue-toad.ts.net/foo/bar.txt works perfectly - it reads /home/mvdan/foo/bar.txt rather than /home/mvdan/foo/foo/bar.txt.

My experience with HTTP proxies is somewhat limited, so I'm not sure whether what I am asking is an anti-pattern of any sort. However, the stripping already appears to happen for path, and intuitively I would find the proxy mode more useful if the stripping happened.

Yet another alternative would be to allow exposing multiple subdomains. For example, instead of exposing this GitHub App backend for testing at https://p14s.blue-toad.ts.net/cue-unity-github-app, I could expose it at https://cue-unity-github-app.p14s.blue-toad.ts.net/. I don't have a strong preference; both are equally easy to identify and remember.

What is the impact of not solving this?

I am able to use Funnel, but it's awkward to use it for more than one HTTP server at the same time - even when those HTTP servers aren't all running at the same time.

Anything else?

No response

[mvdan]

I did take down that /foo example path and file now, by the way, for the sake of not leaving it around forever. But it is hopefully easy to replicate.

[DentonGentry]

I guess the alternative is to run a load balancer like Caddy, and have it strip the paths.

[mvdan]

@DentonGentry I indeed can, but I assume that the main point behind HTTP support in Funnel is convenience, so this tweak would help there :)

[shayne]

You'll run into problems on either side of this issue. However, I feel we should trim the mountPoint in the default (simple) case as running something that would depend on the absolute path would likely be a more advanced use case of Funnel which is better aligned with running Caddy, Traefik, or the like. I don't want to require Caddy to "hook up this OSS web service to Funnel".

[shayne]

Due out in v1.38.3

[mvdan]

Thank you!

[Milo123459]

woo! Thank you!

[Milo123459]

I'm not sure if this is intentional or not, but:

I've added something to serve like that, and when I go to /bandwidth it takes me to the page, and not /bandwidth on the thing I'm proxying. I'm not sure if I'm doing something wrong
So to access /bandwidth on the server, I have to do /bandwidth/bandwidth

[shayne]

Oh, interesting! We don't currently support additional paths for the proxy target, so http://127.0.0.1:3123/bandwidth is treated as http://127.0.0.1:3123. The CLI should've returned an error. I'll file an issue to fix that and will also use that issue to consider if we should add this functionality.

If you want to make serve work this way, the best option today is to use Caddy to configure adding the path to the request.

[Milo123459]

Thanks for the information! In my opinion, this would be extremely useful to have because it could then be even more useful.

[antifuchs]

So, hm, this broke my existing tailscale serve/funnel configuration where I was funneling out two paths relevant to the synapse service on my machine:

{
  "TCP": {
    "8443": {
      "HTTPS": true
    }
  },
  "Web": {
    "gloria.red-acted.ts.net:8443": {
      "Handlers": {
        "/_matrix": {
          "Proxy": "http://127.0.0.1:8448"
        },
        "/_synapse/client": {
          "Proxy": "http://127.0.0.1:8448"
        }
      }
    }
  },
  "AllowFunnel": {
    "gloria.red-acted.ts.net:8443": true
  }
}

Now forwards all requests to /_matrix/client/versions to /client/versions.

Is there any way to get the old (non-path-stripping) behavior back? I tried the following, but it seems to still strip the paths when target paths are specified explicitly:

$ tailscale serve https:8443 /_matrix http://127.0.0.1:8448/_matrix
$ tailscale serve https:8443 /_synapse/client http://127.0.0.1:8448/_synapse/client
$ tailscale serve status
# Funnel on:
#     - https://gloria.red-acted.ts.net:8443

https://gloria.red-acted.ts.net:8443 (Funnel on)
|-- /_matrix         proxy http://127.0.0.1:8448
|-- /_synapse/client proxy http://127.0.0.1:8448

[shayne]

The config you're looking for should work with the recently added support for proxy paths.

Make sure you're on the tailscale version 1.38.4.

$ sudo tailscale serve https /_matrix http://127.0.0.1:3000/_matrix
$ curl https://node.my-tcd.ts.net/_matrix/foo/bar

# receives:
GET /_matrix/foo/bar HTTP/1.1
Host: node.my-tcd.ts.net

Make sure you call sudo tailscale serve ... for each mount point and include the proxy path (i.e., https://127.0.0.1:3000/_matrix). Then, confirm with tailscale serve status --json. You should see the "Web" handlers have the "Proxy" fields with the path added to the end of the URL.

Tip: Start from scratch and clear your serve / Funnel config with $ echo {} | sudo tailscale serve set-raw.

[shayne]

Re-opening to reconsider --strip-prefix option as an advanced use case for serve.

[gromoslaw-kroczka]

@antifuchs - did you manage to set up Matrix Federation trough Tailscale Funnel? I've managed to have working client of Matrix trough it, as you with @shayne described above, but unfortunatelly Funnel is limited to [443, 8443, 10000] ports, where Federation need to connect trough 8448. Do you have any idea how to overcome that?

[bradfitz]

@gromoslaw-kroczka, we could permit 8448 too.

@shayne, objections?

[antifuchs]

@gromoslaw-kroczka:

unfortunatelly Funnel is limited to [443, 8443, 10000] ports, where Federation need to connect trough 8448

You can specify that the federation lives on other ports, using the .well-known delegation method on the http server on your matrix hostname. I have the following on my own matrix domain:

:;    curl https://asf.computer/.well-known/matrix/server ; echo
{"m.server":"matrix.tiger-turtle.ts.net:443"}
:;    curl https://asf.computer/.well-known/matrix/client ; echo
{"m.homeserver":{"base_url":"https://matrix.tiger-turtle.ts.net:443"},"m.identity_server":{"base_url":"https://vector.im"}}

Note that the client service discovery file needs an HTTPS URL and the server URL must be only hostname:port (that caused a matrix outage on my machine for a few hours!); but that's it & you can listen on :443 with funnel. (I use tsnsrv to be the proxy for my homeserver & set up the funneling, but I believe this works just as well with plain tailscale serve)

[gromoslaw-kroczka]

Thank you @antifuchs ! I've missed that part of documentation (only checked somehow SRV DNS record delegation).
Based on your guidelines I've managed to set up Matrix Synapse and exposed it trough Tailscale Funnel :443 with Federation functionality included :D