feat(subsonic): Add support for Reverse Proxy auth - #2557

[crazygolem]

This allows to fully delegate authentication to a third-party service, including the subsonic authentication.
Fixes #2557

While reviewing this PR, please keep in mind that I'm not very familiar with Go and its conventions.
I will address any issue you have regarding the quality/style of the code I produced, if you would be so kind to nudge me in the right direction.

[github-actions]

Download the artifacts for this pull request:

• js-bundle.zip
• binaries.zip

[crazygolem]

I successfully tested using a build from my fork, based on the 0.49.3 release and on master, and with the android clients Ultrasonic and Symfonium.

However I noticed that the navidrome web UI also uses the subsonic /rest/ endpoint and passes a token and salt, which I didn't account for. This happens in ui/src/subsonic/index.js, in the url function.

Since in my productive setup the dummy credentials stored in navidrome don't match the actual ones used by my auth service, and since my auth service doesn't support subsonic's token authentication scheme anyway, this can lead to errors.

Fortunately it is easy enough to work around, as the reverse proxy can simply match on the subsonic query parameter c=NavidromeUI to fall back to the standard authentication (as the web UI sends the session cookies also for requests to the subsonic endpoint), and then with the current code the reverse proxy's user header has priority over the subsonicauth parameters.

It would be nice to be able to disable adding the subsonicauth query parameters from the web UI, but currently I don't have an idea of where to begin with that, and strictly speaking it is not necessary to make this PR useful.

[deluan]

Thanks! I'll take a look and may change the way the UI handles subsonic endpoint authentication.

[crazygolem]

@deluan Is there any chance to get this PR merged soon, even without the UI change? I have been testing it for a few months now using a traefik plugin I wrote to support the feature and with several subsonic clients on android.

As I mentioned previously, adapting the UI calls would be nice, but from my perspective it is not a requirement for the feature, as it targets third-party clients that currently don't work at all with the reverse-proxy setup (unless the user also manages their credentials in navidrome).

Let me know if you are willing to merge like this (or if you plan to have a look at the UI soon) and I will update the PR to make it mergeable again.

[deluan]

I'll take a look again. I remember thinking it has too many changes and the implementation could be simpler, but I have to take another look.

[deluan]

So, AFAICT, with this PR the user would still need to configure their reverse proxy to capture the user and password from the /rest/* endpoints, authenticate the user and pass the username as a Header, right?

Well, not really sure how many users will be willing to do all this, but I'm ok with merging this as long as we can also have some kind of tutorial on how to configure this, including your plugin.

Can you do this and update the PR?

As always, thanks for your efforts!

[crazygolem]

So, AFAICT, with this PR the user would still need to configure their reverse proxy to capture the user and password from the /rest/* endpoints, authenticate the user and pass the username as a Header, right?

Correct, that's the whole point of the change.

If you have a reverse-proxy setup and you already manage authentication on the reverse proxy (i.e. you manage your accounts outside of navidrome for the web app), it will allow to configure ND_ENABLEUSEREDITING=false because you can also use your reverse proxy to perform authentication for the subsonic endpoint too, and users don't need to manage their password in navidrome just for subsonic anymore.

Well, not really sure how many users will be willing to do all this, but I'm ok with merging this as long as we can also have some kind of tutorial on how to configure this, including your plugin.

I suspect that a portion of people that already have a reverse proxy setup could be interested in the change. In principle, my plugin is not necessary as any authentication mechanism supported by both the client and the reverse proxy can be used. For example, Symfonium already supports basicauth, and that's easy to handle on most reverse proxies. My plugin is interesting if you want to also support clients that only know the subsonic authentication scheme. This could also be interesting for opensubsonic client developers, to experiment with new authentication mechanisms (as they don't need to change their subsonic server's code, only configure their reverse proxy).

I can do a guide, no problem. Where were you thinking of having it? In the navidrome documentation on the website?

[bphenriques]

Well, not really sure how many users will be willing to do all this

I am interested in setting up this as I do not want to memorize/manage multiple users/passwords across my different self-hosted apps. I haven't done it yet but it is something I plan to until the end of the year.

I can do a guide, no problem. Where were you thinking of having it? In the navidrome documentation on the website?

I would be interested in this 🙇

[deluan]

I can do a guide, no problem. Where were you thinking of having it? In the navidrome documentation on the website?

We have a small mention to Reverse Proxy in the docs: https://www.navidrome.org/docs/usage/security/#reverse-proxy-authentication

It would be better to maybe have a full page dedicated to it, similar to what we have for Jukebox mode.

[deluan]

Hey @crazygolem, I'm doing some small refactoring in your code, and was wondering: Do we want to fallback to Subsonic API authentication if something goes wrong with the Reverse Proxy authentication? Like, IP does not pass whitelist, user from the header is not found in the DB, etc...?

Edit: NVM, i think this is good for now.
Edit 2: Actually, it is a valid point. If we merge this, users will have to configure their reverse proxy to map the username for subsonic endpoints. This would be a big breaking change!

[crazygolem]

That is a good question.
You actually want to be a bit smarter than just a general fallback to avoid HPP vulnerabilities.

The idea is that if the Remote-User header is present, you don't fallback to anything else, to avoid having the proxy authenticate with one user and navidrome with another. If the header is absent, authentication is performed as before for backwards compatibility.

Note that the test is not whether the header is present, but rather whether it has a value:

navidrome/server/subsonic/middlewares.go

Line 89 in 89312f5

if username := server.UsernameFromReverseProxyHeader(r); username != "" {

So in effect, if the IP is not whitelisted it already falls back on the subsonic authentication. On the other hand, if the header has a value, meaning that a trusted proxy authenticated the user, then it won't fall back and we don't want it to.

I considered making the code more robust by having UsernameFromReverseProxyHeader return an additional value (whether the header was present), but decided against it as 1) it should not happen in principle with a correctly configured proxy; and 2) it follows patterns in the rest of the code.

I also considered moving the logic into its own middleware, but it would have needed more complex changes (I intend to propose something like that in the future, unifying authentication for all endpoints as well and making it configurable to allow hardening a bit your deployment, but I'm not comfortable enough with the codebase to do it right now).

An additional reason it should not break backwards compatibility is that the current reverse-proxy setup requires to bypass proxy authentication for the subsonic endpoint. From the documentation:

If you enable this feature and uses a Subsonic client, you must whitelist the Subsonic API URL, as this authentication method is incompatible with the Subsonic authentication. You will need to whitelist the /rest/* URLs.

So when you upgrade navidrome, if you don't remove the auth bypass on the proxy, the subsonic authentication will kick in as before. And when you remove the auth bypass and forward the Remote-User header, the code path will kick in and ignore the subsonicauth query parameters.

Please let me know if you think of another deployment scenario where backwards compatibility could be broken.

[deluan]

I agree with all your points, I was kind of tired yesterday, and didn't properly analyze your PR and its implications. I can't see any other situation where this would cause backwards compatibility issues.

I also considered moving the logic into its own middleware, but it would have needed more complex changes (I intend to propose something like that in the future, unifying authentication for all endpoints as well and making it configurable to allow hardening a bit your deployment

I started doing this yesterday, but reverted my changes a couple of times due to my misunderstanding. Anyway, I think we should wait on a decision from the OpenSubsonic group about a new authentication mechanism.

What I don't love in this PR is having to call UsernameFromReverseProxyHeader twice for each endpoint call, and some duplicated code re: logs, but I'm ok with it for now, until we refactor the whole authentication system.

Merging the PR now :)

[megatwig]

So when you upgrade navidrome, if you don't remove the auth bypass on the proxy, the subsonic authentication will kick in as before. And when you remove the auth bypass and forward the Remote-User header, the code path will kick in and ignore the subsonicauth query parameters.

This might not hold true for all setups with Caddy. It looks like the forward_auth directive gives you a garbage value if you tell it to copy a header that isn't present, e.g. if you're doing the auth bypass in your SSO and not in the proxy. My setup is Caddy -> Authentik, and my logs are showing msg="API: Invalid login" auth=reverse-proxy error="data not found" username="{http.reverse_proxy.header.X-Authentik-Username}"

I'll look at migrating my clients over to proxy auth tomorrow, but if similar setups are common (I might just be weird) it might be worth changing the logic to fallback to Subsonic auth if the Remote-User doesn't exist.

[crazygolem]

Damn, this makes it a breaking change :(
I did not consider deployments where the header was forwarded and expected to be ignored.

Placeholders in caddy are a bit weird.
There is an open feature request that seems related with a potential fix.
I also found someone else with the exact same issue and a workaround:

# Only set the Remote-User header if forward auth set's X-Authentik-Username
@hasUsername `{http.reverse_proxy.header.X-Authentik-Username} != null`
request_header @hasUsername Remote-User {http.reverse_proxy.header.X-Authentik-Username}

[ghost-of-cerberus]

Wanted to chime in and confirm that I have the same Authentik/Caddy configuration as @megatwig and can confirm that I'm seeing the same issue.

@crazygolem I have tried the workaround but it's not working for me with Navidrome v0.52.0 and Caddy 2.76. Going to give it another go this weekend once I've had a bit more sleep and will update if I'm able to get the workaround...working.

In any case, thanks for the PR and general improvement on this feature. 👍

[crazygolem]

@megatwig @ghost-of-cerberus let me know when you find something that works, it would make sense to add an example for caddy in the documentation PR.

[crazygolem]

@ghost-of-cerberus I managed to make it work in a toy environment.

docker-compose.yml:

version: '3'

services:
  echo:
    image: docker.io/ealen/echo-server:0.9.2
    ports:
      - 80

  caddy:
    image: docker.io/caddy:2.7-alpine
    cap_add:
      - NET_ADMIN
    ports:
      - 127.0.0.1:8080:80
    volumes:
      - $PWD/Caddyfile:/etc/caddy/Caddyfile

Caddyfile:

localhost:80

forward_auth http://echo {
        uri /auth
        header_up X-Echo-Code 200
        #header_up X-Echo-Header X-Authentik-Username:foobar

        #copy_headers {
        #    X-Authentik-Username>Remote-User
        #}
}

# Only set the Remote-User header if forward auth set's X-Authentik-Username
@hasUsername `{http.reverse_proxy.header.X-Authentik-Username} != null`
request_header @hasUsername Remote-User {http.reverse_proxy.header.X-Authentik-Username}

reverse_proxy http://echo

Test with: curl 'localhost:8080/app/'

Response:

{"host":{"hostname":"localhost","ip":"::ffff:10.89.3.15","ips":[]},"http":{"method":"GET","baseUrl":"","originalUrl":"/app/","protocol":"http"},"request":{"params":{"0":"/app/"},"query":{},"cookies":{},"body":{},"headers":{"host":"localhost:8080","user-agent":"curl/8.7.1","accept":"*/*","x-forwarded-for":"10.89.3.15","x-forwarded-host":"localhost:8080","x-forwarded-proto":"http","accept-encoding":"gzip"}},"environment":{"YARN_VERSION":"1.22.19","PATH":"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","container":"podman","NODE_VERSION":"20.11.0","HOME":"/root","HOSTNAME":"af9f8a43967f"}}

Notice that there is no "remote-user" header in the response. If you uncomment the X-Echo-Header line in the Caddyfile and reload the config, the response becomes:

{"host":{"hostname":"localhost","ip":"::ffff:10.89.3.15","ips":[]},"http":{"method":"GET","baseUrl":"","originalUrl":"/app/","protocol":"http"},"request":{"params":{"0":"/app/"},"query":{},"cookies":{},"body":{},"headers":{"host":"localhost:8080","user-agent":"curl/8.7.1","accept":"*/*","remote-user":"foobar","x-forwarded-for":"10.89.3.15","x-forwarded-host":"localhost:8080","x-forwarded-proto":"http","accept-encoding":"gzip"}},"environment":{"YARN_VERSION":"1.22.19","PATH":"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","container":"podman","NODE_VERSION":"20.11.0","HOME":"/root","HOSTNAME":"af9f8a43967f"}}

Note that you have to remove the copy_headers directive for the workaround to work, the workaround outside the forward_auth block replaces it.

[ghost-of-cerberus]

@crazygolem, thanks for continuing to dig on this. I had been testing this as well for the better part of the day and ended up close to your result.
However, with Authentik as my forward auth provider - and whilst this gives a URI that appears to be correct - Symfonium continuously gives an Unable to get media provider version. error using the (Open) Subsonic provider.

*.<domain.tld> <domain.tld> {
	# Navidrome
	@navidrome host <sub.domain.tld>
	handle @navidrome {
		reverse_proxy /outpost.goauthentik.io/* http://authentik:9000
		forward_auth http://authentik:9000 {
			uri /outpost.goauthentik.io/auth/caddy
			import trusted_proxy_list
		}
		# Only set the Remote-User header if forward auth set's X-Authentik-Username
		@hasUsername `{http.reverse_proxy.header.X-Authentik-Username} != null`
		request_header @hasUsername Remote-User {http.reverse_proxy.header.X-Authentik-Username}

		reverse_proxy @geoip_filter navidrome-test:4533 {
			import trusted_proxy_list
			header_up x-Echo-Code 200
		}		
	}
	log {
		hostnames <sub.domain.tld>
		output file /var/log/navidrome_access.log
	}
}

Feels like this is really close and I'm just missing something obvious...

[crazygolem]

Symfonium sends a dummy request with "invalid" credentials (test/test) before sending the real credentials. It expects a proper subsonic error response, which is an HTTP 200 with a subsonic-formatted body containing the error, as well as other metadata such as the server name and version.

Normally this should still work, because with your setup Navidrome is still expected to handle the subsonic authentication as before (we just want to avoid garbage in the Remote-User header if authentication is being bypassed). Do you still have the /rest/* URLs whitelisted in Authentik to bypass authentication?

Also you can remove the X-Echo-Code line from your config, that's useless for Navidrome, it was in my toy environment because it allows to control the echo server's response ^^'

[ghost-of-cerberus]

Thanks for the prompt response.

Normally this should still work, because with your setup Navidrome is still expected to handle the subsonic authentication as before (we just want to avoid garbage in the Remote-User header if authentication is being bypassed). Do you still have the /rest/* URLs whitelisted in Authentik to bypass authentication?

No. I removed that when I began testing the navidrome v0.52.0 build.

Also you can remove the X-Echo-Code line from your config, that's useless for Navidrome, it was in my toy environment because it allows to control the echo server's response ^^'

I've removed that and am still getting the same response from Symfonium when either updating my previous provider or adding a new Subsonic provider. It won't allow me to proceed in either instance.

Additionally, I have tried with the Intercept header authentication auth settings enabled and disabled and made no difference.

Logging in via the web UI with the Caddy configuration works fine as well. Unfortunately, other than Symfonium given that error/message, Caddy, Authentik nor Navidrome are providing anything further that I can see in the logs.

Will try a few additional items and hopefully I can sort out the issue with Symfonium. 👍

[crazygolem]

You're trying to solve two unrelated problems at the same time:

1. Navidrome handles subsonic authentication: the /rest/* URLs must be whitelisted in Authentik and you need the @hasUsername workaround in Caddy to avoid garbage in the Remote-User header.
2. Authentik handles subsonic authentication for clients supporting basicauth: you don't whitelist the /rest/* URLs anymore in Authentik, and enable basicauth (from the docs, that would be the "Intercept header authentication" option). You don't need the @hasUsername workaround because Authentik will always set the header when relevant. Subsonic clients that don't support basicauth will get a weird response they can't handle (a redirect to the login page).

It's quite possible that Symfonium is still bugged and doesn't handle basicauth properly. After I reported the issue, I decided to just write an subsonic adapter plugin for my reverse proxy so I don't have to deal with this :D and to be honest I didn't test anymore with Symfonium's wonky basicauth option. It's possible it still expects a subsonic response on basicauth error, in which case it won't work without extra effort, and I don't know Caddy well enough and Authentik at all to help you.

I suggest to at first use the @hasUsername workaround so you get back a server that works with subsonic clients as before. Then if you still want to invest some time into this, check out DSub (it's available on f-droid for free) as it seems to me that its basicauth implementation is a bit more straightforward.

[ghost-of-cerberus]

Thanks for clearing that bit up @crazygolem, but I think I'm a bit more confused on what should be happening with subsonic authentication and this merged PR. My understanding is that if I have an account which has previously logged into Navidrome via Reverse Proxy Auth, this PR would allow a subsonic client to allow use Reverse Proxy Auth without having to have had previously specifying a password for that user account.

Please could you confirm whether my understanding around this is correct?

If I change my config as you've indicated in1. Navidrome handles subsonic authentication, the auth request is forwarded to Authentik and due to the /rest/* whitelist, is passed to Navidrome which attempts to authenticate the request.

When doing this, all subsonic clients receive an "invalid username / password" error.

If I change my config to 2. Authentik handles subsonic authentication for clients supporting basicauth:, the auth request is forwarded to Authentik, it authenticates the request, and then proxies the request - with headers - to Navidrome. I am also of the understanding that ReverseProxyUserHeader = 'X-Authentik-Username' should still be used in the Navidrome config, as web UI just redirects to the Navidrome login screen otherwise.

Doing this with subsonic clients gives a Connection failure, Unable to get media provider version. or some other seemingly random error.

I've tried this across DSub, subtracks, Tempo and Symfonium but I can log in via the web UI without issue in either of these configurations.

If I have previously logged in via reverse proxy in the web UI, set the user password and use that password in the subsonic clients whilst using v0.51.1, all clients can connect. Same config, or any of the above mentioned configurations, under v0.52.0 give one of the aforementioned errors. I can't even connect with specifying the web UI set user password in the subsonic clients.

What confuses me is that my Caddy, Authentik and Navidrome configs are fairly simplistic all things considered (i.e., Authentik caddy forward auth per docs, Caddy config per docs etc.)

I'll keep trying various combinations and update here if I find something that works.

[crazygolem]

Thanks for clearing that bit up @crazygolem, but I think I'm a bit more confused on what should be happening with subsonic authentication and this merged PR. My understanding is that if I have an account which has previously logged into Navidrome via Reverse Proxy Auth, this PR would allow a subsonic client to allow use Reverse Proxy Auth without having to have had previously specifying a password for that user account.

I'm not sure I understand what you mean, it looks like you want to share an authenticated session between different clients, and that's not what this change is about.

What you can do with the new version, is let your reverse proxy authenticate subsonic clients. Unfortunately, because the subsonic authentication scheme is quite unique and very stupid, nothing else is using it and no reverse proxy or authentication service supports it out of the box.

On the other hand, for web-based subsonic clients (like the Navidrome Web App, which also uses the subsonic API), the reverse proxy can now also authenticate requests to the subsonic API (because the client is web-based, it won't have any issues with redirects, session cookies, etc.). This means that if you don't care about third-party subsonic clients, you can basically disable user management in the Navidrome server, remove the whitelist for the /rest/* URLs in your reverse proxy, and the Navidrome Web App will still work even for requests to the subsonic endpoint.

If I change my config as you've indicated in1. Navidrome handles subsonic authentication, the auth request is forwarded to Authentik and due to the /rest/* whitelist, is passed to Navidrome which attempts to authenticate the request.

This is indeed what is supposed to happen. Now, for this to work, the reverse proxy user header must not be forwarded to navidrome, as the header is no longer ignored (with previous versions, it was ignored if present), and with Caddy you need the @hasUsername workaround to not forward it if Authentik does not perform authentication (otherwise Caddy uses the placeholder itself as value for the header).

If you have configured Navidrome to use Authentik's user header, i.e. ReverseProxyUserHead = 'X-Authentik-Username', then you have to adapt the workaround to send the Authentik header instead of Remote-User:

# Only set the Remote-User header if forward auth set's X-Authentik-Username
@hasUsername `{http.reverse_proxy.header.X-Authentik-Username} != null`
request_header @hasUsername X-Authentik-Username {http.reverse_proxy.header.X-Authentik-Username}

[crazygolem]

@ghost-of-cerberus @megatwig I have updated the reverse proxy documentation with an example for Caddy (you might need to adapt it to your specific deployment, e.g. to forward the non-default X-Authentik-Username if you have configured it in navidrome with ReverseProxyUserHeader).

You can find a preview here. Please comment on the documentation PR if the documentation is wrong or if you find a solution that works better.

[ghost-of-cerberus]

@crazygolem, thank you for the detailed follow up!

I'm not sure I understand what you mean, it looks like you want to share an authenticated session between different clients, and that's not what this change is about.

What you can do with the new version, is let your reverse proxy authenticate subsonic clients. Unfortunately, because the subsonic authentication scheme is quite unique and very stupid, nothing else is using it and no reverse proxy or authentication service supports it out of the box.

Apologies as my ...previously logged in... statement was trying to illustrate a user account that was already created in Navidrome through Remote Proxy Auth via the web portal. Not expecting an active session to be shared.

I think what is causing me issues is that regardless of which implementation I try, no third-party subsonic client will connect unless I set the local password in Navidrome and use that when configuring the server/providers in the apps.

I was expecting to be able to configure a third-party client to where it would take my user credentials, pass it to Authentik where it would validate the credentials and then proxy upstream and access Navidrome as the auth has happened with Authentik; i.e., "poor man's SSO".

I have updated the reverse proxy documentation with an example for Caddy (you might need to adapt it to your specific deployment, e.g. to forward the non-default X-Authentik-Username if you have configured it in navidrome with ReverseProxyUserHeader).

You can find a preview here. Please comment on the navidrome/website#159 if the documentation is wrong or if you find a solution that works better.

Thanks. I'll review and comment, particularly if I find a solution with Authentik as I know it is fairly common with Navidrome users and would help reduce queries and threads like this. 😄

[crazygolem]

I was expecting to be able to configure a third-party client to where it would take my user credentials, pass it to Authentik where it would validate the credentials and then proxy upstream and access Navidrome as the auth has happened with Authentik; i.e., "poor man's SSO".

I'm actually doing this now in my own deployment, but in addition to this change in Navidrome I had to develop an entire plugin for my reverse proxy (it is mentioned at the end of the Traefik example) so that I can properly handle the subsonic authentication scheme. I'm not willing to port the plugin to Caddy myself, so I don't have a nice complete solution for you unfortunately. The Caddy example in the documentation should however support the basicauth-capable subsonic clients (and the response rewriting part should help with Symfonium in particular).

[ghost-of-cerberus]

Thanks, @crazygolem!! That confirms and clears up quite a bit for me.

Greatly appreciate the patience and working with me to sort out why it wasn't behaving as I was expecting!