Keycloak integration as virtual resource (#163)

* Keycloak integration as virtual resource
* Integrate new decision into control-api docs
* Add zone groups sync decision

docs/modules/ROOT/pages/explanation/decisions/keycloak-adapter.adoc

@@ -1,4 +1,7 @@
-= Custom Operator
+= Deprecated: Keycloak Organization Adapter
+
+[INFO]
+This document has been superseded by xref:appuio-cloud:ROOT:explanation/decisions/keycloak-kubernetes-api-integration.adoc[].
 
 == Problem
 
@@ -13,7 +16,7 @@ The Keycloak Organization Adapter maps the in-cluster resources to Keycloak grou
 * Map `Teams` to sub-groups in Keycloak
 ** Create sub-group if Team exits
 ** Manage sub-group members based on Team
-** Import existing sub-groups as Team 
+** Import existing sub-groups as Team
 * Map User resources to users in Keycloak
 ** Set preferences
 ** Import existing Keycloak user as `User`

docs/modules/ROOT/pages/explanation/decisions/keycloak-kubernetes-api-integration.adoc

@@ -0,0 +1,65 @@
+= Keycloak Kubernetes API Integration
+
+== Problem
+
+In order to allow users to manage access to their organizations we need an adapter between Keycloak and the Kubernetes API to manage users, groups, and teams in Keycloak.
+
+The first user management implementation was done using a controller und custom resource definitions (CRDs).
+This implementation effectively had two sources of truth: Keycloak and the Kubernetes API; and tried to keep them in sync.
+Over time those two sources of truth diverged and we had to deal with more and more sync inconsistencies.
+
+We had users try to use the Kubernetes API as an interface to their user management needs, but this was abandoned due to various inconsistencies.
+
+We now want to revisit this problem and come up with a new solution without the need to keep two sources of truth in sync.
+
+.Goals
+
+* Map `Organization` / `OrganizationMembers` resource to groups in Keycloak
+* Map `Teams` to sub-groups in Keycloak
+* Map User resources (their preferences) to users in Keycloak
+
+.Non-Goals
+
+* Operational management of Keycloak (for example to create and manage realms)
+
+== Proposals
+
+=== Option 1: Create a virtual resource for OrganizationMembers, Team, and User
+
+We create a virtual resource for OrganizationMembers, Team, and User that is backed by Keycloak.
+
+.Pro
+
+* We can avoid the sync inconsistencies
+* We can use the Kubernetes API as an interface to manage users in Keycloak
+
+.Con
+
+* Upfront investment to create a new implementation
+* Potential performance issues
+* API is down when Keycloak is down
+
+=== Option 2: Keep and improve controller
+
+We keep the existing controller and improve it and resolve sync inconsistencies.
+
+.Pro
+
+* We already have a working implementation
+* Keycloak can be unavailable and we can still manage users in Kubernetes
+
+.Con
+
+* Doubtful we can ever solve the sync inconsistencies
+* We might never move away from the two sources of truth
+
+== Decision
+
+Create a virtual resource for OrganizationMembers, Team, and User.
+
+image::idp-integration.svg[VSHN IDP integration,400]
+
+== Rationale
+
+We want to avoid the sync inconsistencies and the need to keep two sources of truth in sync.
+We're doubtful we can ever solve the sync inconsistencies with a controller.

docs/modules/ROOT/pages/explanation/decisions/keycloak-zone-group-sync.adoc

@@ -0,0 +1,63 @@
+= Keycloak Zone Group sync
+
+== Problem
+
+Currently we use https://github.com/appuio/group-sync-operator[group-sync-operator] to sync groups from Keycloak to Kubernetes.
+We forked the upstream project to add support for nested groups.
+
+The controller is slow, not very reliable, and users often have to wait for the sync in the zone to complete.
+We have no way of showing the status of the sync to the user.
+Hidden wait times lead to user confusion and we had support cases because of that.
+
+.Goals
+
+* Instantly sync groups from Keycloak to the APPUiO zone
+* Provide a good initial experience for the user
+
+.Non-Goals
+
+* Provide additional OpenShift group management features
+
+== Proposals
+
+=== Option 1: Provide groups in OIDC token
+
+We build a custom Keycloak OIDC token mapper that adds the groups to the token.
+The mapper would join the groups with the selected separator, trim the "organization" root group, and add the groups to the token.
+
+.Pro
+
+* Every login sync the groups from Keycloak to the zone
+* The sync is in the request path and the user gets notified on errors
+
+.Con
+
+* User needs to re-login to sync the groups after they changed
+* Writing and managing a custom Keycloak OIDC token mapper
+
+=== Option 2: Agent on the zone watches control-api
+
+We build an agent that watches the control-api for changes and updates the groups in the zone.
+
+.Pro
+
+* Users do not have to re-login to sync the groups.
+* All boilerplate is done, we already implemented similar controllers in the zone agent.
+
+.Con
+
+* The errors do not happen in the request path and might be less visible to the user and us
+** Needs better instrumentation
+
+== Decision
+
+We add a controller to the agent on the zone to sync groups.
+
+== Rationale
+
+Requiring the user to re-login to sync the groups is not a good user experience.
+It is less work to implement and maintain than a custom Keycloak OIDC token mapper.
+
+== Resources
+
+- https://www.baeldung.com/keycloak-custom-protocol-mapper[Keycloak Custom Protocol Mapper]

docs/modules/ROOT/pages/references/architecture/control-api-org-adapter.adoc

@@ -48,8 +48,8 @@ The group membership information is then present as a claim in the OIDC token pr
 As long as the Identity Provider is not custom built, it is very unlikely to be able to interface with the {controlapi}.
 Providing this interface is the job of the Organization Adapter.
 
-This adapter **should** be implemented as https://kubernetes.io/docs/concepts/extend-kubernetes/operator/[Kubernetes Operator^] that interfaces with the four {controlapi} resources related to user management: `Organizations` and `OrganizationMembers`, `Teams`, and `Users`.
-
+The adapter **may** be implemented as a Kubernetes https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/[aggregated API server] that interfaces with the {controlapi} and the IdP.
+The adapter **may** be implemented as https://kubernetes.io/docs/concepts/extend-kubernetes/operator/[Kubernetes Operator^] that interfaces with the four {controlapi} resources related to user management: `Organizations` and `OrganizationMembers`, `Teams`, and `Users`.
 
 [NOTE]
 Some of the following requirements **may** be implemented by or through the IdP itself or by a third system.
@@ -68,13 +68,15 @@ For each `Organization` and the corresponding `OrganizationMembers` resource
 The adapter **should** provide the option to import existing groups from the IdP.
 If this feature is provided
 
+[NOTE]
+Imports might not be required if the resources are virtual and the IdP is the authoritative source.
+
 * The adapter **must** create `Organization` resources for groups in the IdP that represent organizations if a corresponding `Organization` does not exist.
 * The adapter **must** update the corresponding `OrganizationMembers` resource of a newly created `Organization` to represent the members of the group in the IdP.
 * The adapter **must not** edit existing `Organization` or `OrganizationMembers` resources as part of an import.
 * The adapter **should** provide means to set RBAC for imported organizations, so that one or more users can manage the `Organization` in the {controlapi}.
 * The adapter **may** mark imported organizations as externally managed.
 
-
 The adapter **may** provide the option to mark one or more `Organizations` as to be managed externally by the IdP.
 This means the `Organization` and `OrganizationMembers` resources are read-only and updated to the state in the external system.
 If this feature is provided
@@ -113,13 +115,15 @@ For each `Team` resource
 The adapter **should** provide the option to import existing groups from the IdP.
 If this feature is provided
 
+[NOTE]
+Imports might not be required if the resources are virtual and the IdP is the authoritative source.
+
 * The adapter **must** create `Team` resources for groups in the IdP that represent teams if a corresponding `Team` does not exist.
 * The created `Team` **must** reference all the members of the group in the IdP.
 * The created `Team` **must** be in the `Namespace` of the organization containing the team.
 * The adapter **must not** edit existing `Team` resources as part of an import.
 * The adapter **may** mark imported teams as externally managed.
 
-
 The adapter **may** provide the option to mark one or more `Teams` as to be managed externally by the IdP.
 This means the `Team` resource is read-only and updated to the state in the external system.
 If this feature is provided
@@ -132,14 +136,31 @@ If this feature is provided
 
 .Organization Adapter at VSHN
 ****
+VSHN started out with an operator but VSHN grew increasingly unhappy with the sync complexity of having two sources of truth.
+
+We are now in the process of replacing the operator with a Kubernetes aggregated API server.
+This allows us to reduce sync complexity and to provide a more consistent user experience.
+
+image::idp-integration.svg[VSHN IDP integration,400]
+
+The new adapter will live in the https://github.com/appuio/control-api[control-api^] repository and will be a part of the control-api deployment.
+Different IDPs can be supported by implementing a https://github.com/appuio/control-api/blob/e4164a17097e15a1ac91d9cce969d07c64861344/apiserver/billing/odoostorage/odoo/odoo.go#L10[Go interface] like for the ERP integration.
+
+The first resource that will be virtualized will be the `User` resource.
+We see the biggest sync skew there and it is the most important resource for the user experience.
+
 The VSHN-managed {product} instance uses the https://github.com/vshn/appuio-keycloak-adapter[APPUiO Keycloak Adapter^].
 
-The adapter is a Kubernetes Operator that creates and manages Keycloak groups and sub-groups that correlate to `Organization` and `Team` resources.
-A per {zone} deployed https://github.com/redhat-cop/group-sync-operator[Group Sync Operator^] then synchronizes group memberships from the Keycloak groups to OpenShift.
+We're currently using https://github.com/redhat-cop/group-sync-operator[Group Sync Operator^] to sync groups from Keycloak to OpenShift.
+With the implementation of UsageProfiles we now have good experience with multi event-source reconciliation and we'll let the APPUiO Cloud Agent sync the groups from the control-api to the zone.
+The addition of watching changes should improve the user experience by making most portal operations instant.
 
 The adapter also imports Keycloak users as `User` resources.
 Changes to the `User` resource's `spec` is added to the Keycloak user's attributes.
 These attributes are then synced to the {zone}s by the https://github.com/appuio/keycloak-attribute-sync-controller[Attribute Sync Controller^].
 
+We're currently using https://github.com/appuio/keycloak-attribute-sync-controller[Attribute Sync Controller^] to sync the default organization to the zones.
+We'll let the APPUiO Cloud Agent sync the default organization to the zones and implement watching changes in the control API to improve the user experience.
+
 The Keycloak adapter also supports periodic imports of new Keycloak groups as `Organizations`.
 ****

docs/modules/ROOT/pages/references/architecture/control-api-org.adoc

@@ -158,7 +158,7 @@ status:
   - name: kate.demo
   - name: peter.muster
 ----
-<1> References to one or more xref:references/architecture/control-api-user.adoc[`User`] resource. 
+<1> References to one or more xref:references/architecture/control-api-user.adoc[`User`] resource.
     The `name` field must match `metadata.name` of an existing `User` resource.
-<2> This is resolved by the xref:explanation/system/details-adapters.adoc[adapter]. 
+<2> This is resolved by the xref:explanation/system/details-adapters.adoc[adapter].
     May only contain a subset of users if the adapter is unable to add some users.

docs/modules/ROOT/partials/nav.adoc

@@ -148,8 +148,10 @@
 ** xref:appuio-cloud:ROOT:explanation/decisions/efk-openshift-logging.adoc[RHOL EFK Logging Stack]
 ** xref:appuio-cloud:ROOT:explanation/decisions/billing-etl.adoc[]
 ** xref:appuio-cloud:ROOT:explanation/decisions/billing-entity-virtual-resource.adoc[BillingEntity as Virtual Resource]
-** xref:appuio-cloud:ROOT:explanation/decisions/keycloak-adapter.adoc[Keycloak Organization Adapter]
+** xref:appuio-cloud:ROOT:explanation/decisions/keycloak-kubernetes-api-integration.adoc[]
+** xref:appuio-cloud:ROOT:explanation/decisions/keycloak-adapter.adoc[]
 ** xref:appuio-cloud:ROOT:explanation/decisions/no_rbac_creation.adoc[]
 ** xref:appuio-cloud:ROOT:explanation/decisions/control-api-interaction-with-zones.adoc[]
 ** xref:appuio-cloud:ROOT:explanation/decisions/control-api-odoo16-api.adoc[]
+** xref:appuio-cloud:ROOT:explanation/decisions/keycloak-zone-group-sync.adoc[]
 ** xref:appuio-cloud:ROOT:explanation/decisions/night-maintenance.adoc[]