Sync identity provider groups and user deactivations from Active Directory using SCIM

[rosszurowski]

Several users have written in asking for the ability to sync user groups from their identity provider. This is really useful for providers like Okta, which are designed to provide this configuration to apps. We should save our users from this manual work and import it for them, probably using SCIM.

Front conversations

[jack1902]

@rosszurowski wondering if this would be the start of breaking the ACL down into useable pieces?

Currently i see the syncing of groups and the ACL itself as two seperate pieces (even though currently, it is all defined within one ACL file)

Wondering if it would be possible to have the split so that, Groups and their members can be synced without having to change the ACL itself? For example, i might want to control what groups have access to other groups but not care about who is in those groups as that is handled by Okta. Currently, i'd have to ensure that i have the latest ACL from the API before i make any updates to the ACL itself so that the Groups section isn't completely wiped out by one update call

[rosszurowski]

@jack1902 good point. I'm not sure the exact form this would take. @apenwarr has thought a lot about our ACL file and may have opinions here.

In general, I think we do want to keep the single ACL file as a "source of truth" for what's allowed on a domain so it can be updated via API, version-controlled, etc. But even if a single file is the source of truth, that doesn't mean we can't provide more convenient interfaces on top of it. We may, for example, introduce separate API endpoints to allow editing just "ACLs" without touching the "Groups" or "Hosts". Similarly, we'll probably introduce a UI for managing user groups that shouldn't affect other sections.

[apenwarr]

When we get the auto-syncing working, the plan is to use Okta SCIM to sync into something like an autogroup: section that can be referred to from the ACL file, but not actually be listed in the ACL file. Then the existing statically-defined groups can be used a lot less often, and thus the ACL file itself will change a lot less often.

Separately, as we're working on an API for modifying ACLs, we should have it accept an atomic "overwrite-if-equals" kind of operation, which will reject changes to the ACLs unless the change request also includes eg. the ETag of the ACLs that are supposed to be there before the change. If the ETag doesn't match, then someone else got there before you, sorry, try again. Like an opportunistic transaction in a database. That should safely cover all cases, although it would be a little annoying for a client to implement sometimes.

[DentonGentry]

Summarizing some internal discussions about creation of autogroups using SCIM over the last several months:

• Until something automatic is working, the API can also be used to automate rule creation: http://github.com/tailscale/tailscale/blob/main/api.md
• Google stopped accepting SCIM integrations in 2019, no solution for Google likely in the near future (though Okta can provide SCIM with Google as a backend)
• Okta and AzureAD are the two highest priority

The current release in the field is 1.8, released last week. Odd numbers are development releases, so 1.9 is what we're working with right now and will become a 1.10 release in late June. Work on synchronizing ACL groups is not currently planned for 1.10 but would be in scope for releases after that.

We're likely to begin work on SCIM to synchronize to AzureAD first, with Okta coming slightly later.

[DentonGentry]

For those following along, work on SCIM for Okta and Azure is underway.

[DentonGentry]

SCIM design review happened this week.

[jpluscplusm]

SCIM design review happened this week

May I ask for an update as to where SCIM support is currently sitting in Tailscale's backlog?

[DentonGentry]

We have three engineers currently working on it. Okta is being worked on first, then Azure. Delivery isn't imminent, but development is underway.

[hrzbrg]

Are there updates on this topic? SCIM and groups would significantly reduce the manual administrative overhead for us.

[DentonGentry]

Group and user suspension synchronization for Okta exited Alpha yesterday: https://tailscale.com/blog/sync-okta-groups/

It is now available in Beta for anyone to try. Please open new issues to report problems you encounter, don't add it here. This issue tracks the overall effort.

[hopinto]

Would OneLogin SCIM support make sense from a business perspective?

[DentonGentry]

I would expect to eventually add OneLogin support as well if it is possible to do so, but expect Azure to come next before Onelogin work would start.

[owenda7]

Are there any time estimates for AzureAD SCIM Beta release?

[DentonGentry]

I promise that if progress is made on this, we'll update the bug. Much of the work done on SCIM for Okta will be directly applicable.

[jsommr]

Shouldn't an open standard like SCIM just work with various providers? How come a server must have knowledge about which provider is provisioning users (besides a shared key of some sort)?

[DentonGentry]

Having an implementation working with one provider helps a great deal, but rarely works immediately until the standard has become extremely widely implemented (which SCIM has not). There is debugging, and adapting to the interpretations each provider makes of what the standard says to do.

[josedev-union]

Could you share the status of AzureAD SCIM release?

[DentonGentry]

When we have progress to report it will be posted here.
The internal work being done is not to the point of posting here yet.

[mayakacz]

Development on user & group provisioning (SCIM) for Azure AD is underway. When we have further progress to report, we will post it here.

[pcibraro]

We have an existing subscription In Tailscale, and we would like to try to use the SCiM APIs directly without using Azure AD or Okta. We have a customer adapter we wrote for our identity provider. Are those APIs available for any customer to use?

[DentonGentry]

There is an API to update the ACL file where groups are defined: https://github.com/tailscale/tailscale/blob/main/api.md#update-policy-file

The flow would be:

1. Fetch the current Tailscale ACLs.
2. Apply the updated group definitions. HuJSON has a Patch operation which can do this without needing to fully parse the original ACLs.
3. POST back to the API to write the new ACLs using If-Match to determine if there were any intervening writes.

There is an example of doing this in https://github.com/DentonGentry/tailscale-aws-host-acl-updater, in acls.go.

[pcibraro]

Yes, I am familiar with the ACL Api. We were planning to use that one until I saw you had a beta version for SCIM. The thing with the ACL Api is that we require writing custom code just to integrate with Tailscale when we had a SCIM implementation already in place for pushing data.

[DentonGentry]

Every identity provider we've worked with has required substantial effort from us. SCIM is a large spec, of which every provider implements a different subset. The Patch operation has required new work as each IdP uses it quite differently, their subset of attributes is different each time, and their interpretation of what Suspended and other states mean has varied greatly. As such, we don't make the SCIM API publicly available. It has required us to do substantial custom work each time.

The mechanism we have available is https://github.com/tailscale/tailscale/blob/main/api.md#update-policy-file

[pcibraro]

Sounds good! I understand the issue. Thanks.

[plett]

Good news! Syncing Azure AD users and groups to Tailscale went into beta last week: https://tailscale.com/kb/1249/sso-azure-ad-scim/

I have been subscribed to this issue waiting for the promised updates on progress from Tailscale staff but then spotted this KB article in their changelog feed, so thought I would let everyone else who has been watching this issue know about it.

[TheMrDrProf]

Curious if anyone has got this setup and working. Super happy to see it implemented, however groups are not syncing. We configured precisely following the Tailscale docs, other than we did not create the Tailscale Enterprise App since it already existed as our auth method to Tailscale; otherwise configured precisely following docs.

SCIM in Azure AD shows provisioning cycles running. I only have 5 groups directly assigned to the Tailscale app for sync. There are 3 users that are members of some number of these groups. SCIM provisioned the 3 users (which already existed in Tailscale), but is not syncing any groups. Reviewing the provisioning logs, all groups that are queued for sync are skipped:

On Tailscale side, it shows successful syncs, but no groups:

Anyone able to provide guidance on this?

[hazcod]

@TheMrDrProf we have the exact same situation and issue, no resolution from support yet.

[crawshaw]

Hello. We have identified an issue with Azure group sync and are working with Microsoft to get it resolved. I don’t have an ETA yet, but we will update this thread when we know more.

[VictorD]

In case it helps the troubleshooting, group sync started working for us after I altered the JSON schema in our Enterprise Application to include a group section following the "urn:ietf:params:scim:schemas:core:2.0:Group" spec

Probably not the way it is meant to be done, but seems like the problem is with the default schema

edit: disclaimer use at own risk, but here's a gist of what I think I changed back then... was a few weeks ago
https://gist.github.com/VictorD/e9ddf3c1bdeaf4ae12cf596a3dc03e9b

[deandre]

To expand on @crawshaw's comment, there was a bug in how Azure AD enabled group sync for Tailscale, which uncovered a larger group sync issue for Azure AD. Azure AD is working on this issue, and specifically republishing the Tailscale app. We expect to confirm a fix this week, and republish the app in the coming ~2 weeks.

When the app is republished, group sync should be enabled by default and start working for any existing installations of the app.

Meanwhile, if necessary, you can install Tailscale as a custom app for group sync. This is not recommended as it will need to be replaced once the updated app is published.

[TheMrDrProf]

This appears to now be working for me. Groups have appeared in the SCIM settings in Azure AD:

And I now have groups available in Tailscale: (Ignore the [WARP], we came from Cloudflare)