Self Hosting and OIDC: hello from the rabbithole

[NoArmyKnife]

Prologue

After spending some days trying to self host Netbird with proper SSO (emphasis is important), I started to browse through the different Issues and Discussions of the project's Github page when I found other fellow self-hosters in distress: #5143

This discussion is my modest attempt at giving back to the community by summarizing the information I've gathered along my journey.

Before diving in, please note a few things:

• this discussion is neither ranting nor criticism of the incredible work that has been done so far by the Netbird team
• I am not a security expert, and even less a golang developer. I'm just a 8-YoE DevOps Engineer (with a solid development background) trying to setup a cool piece of software and who met some bumps on the road
• as a consequence of the previous bullet point, I might have overlooked/not seen/not understood certain subtelties of the project ; if so, please do tell

That being said (coconuts in Barbados), it's time to go for a ride !

TOC

1. Testing setup

2. What works and what does not

3. Suggestions
   3.1. Workarounds for a quick setup
   3.2. Quick-win fixes
   3.3. Long-term fixes for ensured stability

4. Closing thoughts

1. Testing Setup

For this investigation, I tested both the "legacy" (5-container stack) setup as well as the combined one, using a mix of IdPs and authentication flows:

• Netbird versions: v0.70.5 and v0.71.0
• Deployment mode: Combined (i.e. with the Dex Embedded IdP) and "Legacy" (i.e. Authz/Authn is fully handled by my own, "external" IdP) behind an already running Nginx reverse proxy
• IdPs: Authelia and Authentik.
• Flows:
  • Authorization Code Flow (Dashboard)
  • Device Authorization Flow (Netbird CLI on a Raspberry Pi, version matching the server's)
  • PKCE Flow (Android App, v0.4.2).

2. What Works and What Does Not

The biggest pain points revolve around the interaction between the Embedded IdP, group synchronization, and Single LogOut (SLO).

Major Issues Found

• Group Syncing: Netbird trying to synchronize groups from the "groups" claim in the ID token makes the feature unreliable because the groups claim is not a stable ID Token claim and therefore should not be blindly expected to be present. The solution would be to perform a request to the userinfo endpoint, but even then it would still be an issue as-is because the groups claim is not an OIDC Standard Claim.

• Single LogOut (SLO): In combined mode (with the Dex Embedded IdP), the logout process is flawed. The IdP seen by Netbird is effectively the embedded one, so it looks like the logout only happens from the Embedded IdP and not the "actual" upstream IdP. The "proof" of this is that when I clicked "Logout" in Netbird, I was redirected to Netbird's home page but:

  • I could simply click on "Login with " and be instantly logged in (i.e. not prompted for my credentials)
  • I could go to any other SSO-protected resource and see that I was still logged in

I'm walking on eggshells here, but this might be the reason why (from the Dex Source code):

// LogoutURL returns the upstream OIDC provider's end_session_endpoint URL.
// Per the OIDC RP-Initiated Logout spec, the post_logout_redirect_uri parameter
// tells the upstream where to redirect after logout.
func (c *oidcConnector) LogoutURL(_ context.Context, _ []byte, postLogoutRedirectURI string) (string, error) {
	if c.endSessionURL == "" {
		return "", nil
	}

	u, err := url.Parse(c.endSessionURL)
	if err != nil {
		return "", fmt.Errorf("oidc: failed to parse end_session_endpoint: %v", err)
	}

	q := u.Query()
	if postLogoutRedirectURI != "" {
		q.Set("post_logout_redirect_uri", postLogoutRedirectURI)
		q.Set("client_id", c.oauth2Config.ClientID)
	}
	u.RawQuery = q.Encode()

	return u.String(), nil
}

// HandleLogoutCallback is a no-op for OIDC. The end_session_endpoint simply
// redirects back without a structured response to validate.
func (c *oidcConnector) HandleLogoutCallback(_ context.Context, _ *http.Request) error {
	return nil
}

Note: SLO works fine in the legacy "split" mode, provided RP-initiated logout is enabled in the IdP. I tested it myself with Authentik (make sure you follow the instructions here).

• PKCE Flow Failure (from Android App) in Legacy mode: For some mysterious reason, the PKCE flow fails to create the callback server on the phone when used with Authelia (it works well when using Authentik, that's what makes it weird). I suspect something is going wrong around here but I could not really understand what.

// WaitToken waits for the OAuth token in the PKCE Authorization Flow.
// It starts an HTTP server to receive the OAuth token callback and waits for the token or an error.
// Once the token is received, it is converted to TokenInfo and validated before returning.
// The method creates a timeout context internally based on info.ExpiresIn.
func (p *PKCEAuthorizationFlow) WaitToken(ctx context.Context, info AuthFlowInfo) (TokenInfo, error) {
// ...
	go p.startServer(server, tokenChan, errChan)
// ...
}

What Worked (and how to improve it)

• Combined Mode: I managed to successfully connect as users from my external IdP with both Authelia and Authentik, provided a few tweaks to the Netbird source code (in idp/dex/connector.go):

switch cfg.Type {
        case "zitadel":
                oidcConfig["getUserInfo"] = true
        case "authentik":
                oidcConfig["getUserInfo"] = true
                oidcConfig["scopes"] = []string{"openid", "profile", "email", "groups"}
        case "oidc":
                oidcConfig["getUserInfo"] = true
                oidcConfig["scopes"] = []string{"openid", "profile", "email", "groups"}
        // ...
}

This effectively forces Dex to hit the upstream IdP's userinfo endpoint and inject the needed claims into the idToken, which is what the Netbird Dashboard expects.

Personal note: I believe it is a fairly sensible default to set the "getUserInfo" Dex config to true for every provider type since the claims wanted by Netbird (name and groups for now, but could be others in the future) are not default stable ID Token claims, and therefore should not be systematically expected by an OIDC Client.

• Legacy Mode: provided the management.json contains all the necessary information, and if we only talk about Authorization/Authentication, the legacy deployment mode works with both Authelia and Authentik. However, if you decide to go this route, please consider the following caveats:

  • For Authelia

    • in order for Groups sync to work, you MUST configure your OIDC client to include the groups claim in the idToken. This is done with a claim policy
    • as explained earlier, Authentication from the Android App will not work, I can't understand why
    • in order to use the Device Code Authorization Flow (used by Linux CLI for example), you will need a second OIDC client in Authelia because the current configuration will simply not allow you to have a single client that matches all the criteria for it to work
    • the value of AUTH_REDIRECT_URI will need to be changed to a fragment-less value (this is because Authelia actually follows the OIDC specification, thus enforcing RFC 6749 and RFC3989 during redirect URI verification)
    • since Authelia does not yet support RP-initiated logout, logging out from Netbird will not invalidate the user's Authelia session but only redirect them to the Authelia portal (from where they can logout alright)

  • For Authentik

    • in order for Groups sync to work, your provider MUST be configured to include the groups claim in the idToken. See this Authentik documentation page
    • if not already done, a device code flow should be created if you want to be able to use this Authorization flow (tested with Linux CLI)

I will try to provide minimal configuration snippets for those willing to test this.

3. Suggestions

Workaround for a Quick Setup

For those who want a setup that just works (i.e. no fixes/tweaks needed in the Netbird source code) with:

• external IdP
• Groups Sync
• Single LogOut

Then I should recommend using the "legacy" 5-container stack deployment mode with Authentik (remember to follow the instructions here).

For those who really want to use Authelia, I would recommend the "combined" mode deployment but you would need to choose the Okta or Pocketid type (instead of OIDC Generic) in the Identity Provider creation modal.

In order to improve the project toward more strict adoption of the OIDC protocol (which needs more strict respect of the OIDC specification), I propose the following fixes:

Quick-Win Fixes

1. Dashboard Callback URLs: Update the Dashboard's default callback URLs (/#callback and /#silent-callback) to be RFC compliant by removing the fragment (#).
2. Group retrieval: The system needs a standardized way to request groups. Instead of relying on hardcoded scopes, we should:
   • Allow groups retrieval per IdP via a switch/toggle in the IdP creation/update modal (in addition to a field allowing to customize the name of the claim)
   • Ideally, this should be applied across all connector types (see idp/dex/connector.go#L230).

Long-Term Fixes for Ensured Stability

1. Alllowing to disable the Embedded IdP: For people who start their self hosting journey with Netbird, the embedded IdP is a formidable idea. Netbird provides them with what they came for (a network mesh) and a turnkey IdP. This is wonderful. However, from what I could see, there are also lots of users coming to Netbird with an already working OIDC-compliant IdP. For those users, having to go through the hassle of making Dex work with their Idp (and in the worse cases, having to tweak it for that sole purpose) is a huge point of friction. This is why I would recommend this feature, implemented with a proper fallback mechanism. If the Embedded IdP is disabled, the configuration should gracefully fall back to reading relevant settings from the legacy files (e.g. management.json), or better yet, bring those settings up to the combined config.yaml.
   Note: it is an even more important topic knowing that the Dex OIDC connector is a beta feature

2. User information source: in order to take a step forward towards OIDC compliance, the profile information (name, email, groups, etc.) should be fetched from the userinfo endpoint, instead of the ID Token as currently done (https://github.com/netbirdio/netbird/blob/main/shared/auth/jwt/extractor.go#L93)

3. SLO Fix: The SLO mechanism needs a complete overhaul to ensure the user is logged out from all connected services, not just the Embedded IdP (beware this might not be possible if Dex does not relay the logout request through front/back-chanel logout ; I don't know for sure)

4. Closing Thoughts

Overall, the current authentication implementation is already pretty solid, and OIDC is a really complex standard with lots of specifications so props to you for initiating the work. I still believe that we - the OSS community - should do our best to implement it as close as possible to the specification when we decide to go this route. It ensures both stability and security, as well as increased confidence from the users to see that things are done according to established standards.
For those interested, an Authelia core member wrote a very interesting blog article on this topic: https://www.authelia.com/blog/technical-openid-connect-1.0-nuances/

Whatever the main development team decides to do with these observations, I'd be more than happy to perform tests and/or small code fixes that have limited impact (remember, not a golang developer, even though I'm a fast learner).

In any case thank you for developing Netbird !