Merge pull request #154 from appuio/architecture/usage-profiles

Add architecture documentation for usage profiles

docs/modules/ROOT/pages/references/architecture/control-api-usage-profile.adoc

@@ -0,0 +1,133 @@
+= APPUiO Control API: UsageProfile
+
+TIP: This resource implements the xref:appuio-cloud:ROOT:references/quality-requirements/performance/resource-quota.adoc[] and xref:appuio-cloud:ROOT:references/quality-requirements/performance/ns-quota.adoc[] requirements.
+
+The `UsageProfile` encapsulates a set of quotas and policies which are applied to a namespace or a set of namespaces.
+Every organization has a `UsageProfile` assigned.
+When a new organization is created, a default `UsageProfile` is assigned.
+Depending on whether the new organization's `BillingEntity` has been validated, a different default `UsageProfile` is assigned.
+
+
+== Object
+
+.CRD based
+[source,yaml]
+----
+apiVersion: appuio.io/v1
+kind: UsageProfile
+metadata:
+  name: restricted <1>
+spec:
+  namespaceCount: 50 <2>
+  resources: <3>
+    organization-compute: <4>
+      apiVersion: v1
+      kind: ResourceQuota
+      spec:
+        hard:
+          limits.cpu: "8"
+          limits.memory: 20Gi
+          pods: "45"
+          requests.cpu: "4"
+          requests.memory: 4Gi
+        scopes:
+          - NotTerminating
+    organization-compute-terminating:
+      apiVersion: v1
+      kind: ResourceQuota
+      metadata: <5>
+        labels:
+          foo: bar
+      spec:
+        hard:
+          limits.cpu: "4"
+          limits.memory: 4Gi
+          pods: "5"
+          requests.cpu: 500m
+          requests.memory: 2Gi
+        scopes:
+        - Terminating
+    organization-objects:
+      apiVersion: v1
+      kind: ResourceQuota
+      spec:
+        hard:
+          cephfs-fspool-cluster.storageclass.storage.k8s.io/requests.storage: 25Gi
+          count/configmaps: "150"
+          count/jobs.batch: "150"
+          count/replicationcontrollers: "100"
+          count/secrets: "150"
+          count/services: "20"
+          count/services.loadbalancers: "0"
+          count/services.nodeports: "0"
+          limits.ephemeral-storage: 500Mi
+          localblock-storage.storageclass.storage.k8s.io/persistentvolumeclaims: "0"
+          openshift.io/imagestreams: "20"
+          openshift.io/imagestreamtags: "50"
+          persistentvolumeclaims: "10"
+          rbd-storagepool-cluster.storageclass.storage.k8s.io/requests.storage: 25Gi
+          requests.ephemeral-storage: 250Mi
+          requests.storage: 1000Gi
+    deny-egress-tcp-25: <6>
+      apiVersion: networking.k8s.io/v1
+      kind: NetworkPolicy
+      metadata:
+        annotations:
+          description: "Deny egress traffic to all hosts on ports 25"
+      spec:
+        podSelector: {}
+        egress:
+          - to:
+              - ipBlock:
+                  cidr: 0.0.0.0/0
+            ports:
+              - protocol: TCP
+                port: 25
+        policyTypes:
+          - Egress
+----
+<1> The name of the usage profile, used to reference it from other resources (for example `Organization`).
+<2> `spec.namespaceCount` defines how many namespaces an organization with this usage profile can create per zone.
+<3> `spec.resources` is a map whose values are expected to be Kubernetes manifests.
+This field defines the set of resources which are created in each namespace for which the usage profile is applied.
+<4> The keys are injected into the associated value as `metadata.name`.
+Setting `metadata.name` in a value has no effect.
+<5> Additional metadata, such as labels or annotations, can be defined in the values.
+<6> There's no restriction on the kinds of manifests that can be defined as part of a `UsageProfile`.
+
+.BillingEntity resource
+[source,yaml]
+----
+apiVersion: billing.appuio.io/v1 <1>
+kind: BillingEntity
+spec:
+  usageProfileRef: restricted <2>
+----
+<1> Only relevant fields are shown for brevity
+<2> A `BillingEntity` references a `UsageProfile`.
+This `UsageProfile` is the default usage profile for all organizations associated with the billing entity.
+
+.Organization resource
+[source,yaml]
+----
+apiVersion: organization.appuio.io/v1 <1>
+kind: Organization
+spec:
+  billingEntityRef: be-1234 <2>
+  usageProfileRef: default <3>
+status:
+  usageProfileRef: default <4>
+----
+<1> Only relevant fields are shown for brevity
+<2> An `Organization` always references a billing entity.
+<3> An `Organization` can optionally explicitly reference a `UsageProfile`.
+<4> The {controlapi} updates `status.usageProfileRef` with the usage profile that should be applied.
+When field `spec.usageProfileRef` of the `Organization` is set, the value of that field is propagated to field `status.usageProfileRef`.
+Otherwise, the referenced billing entity's field `spec.usageProfileRef` is propagated to field `status.usageProfileRef`.
+This field defines the base set of quotas and policies which are applied to all namespaces belonging to this organization.
+
+== Access control
+
+This resource is cluster-scoped.
+No RBAC rules are generated automatically.
+A cluster admin must allow users to view or edit `UsageProfile` resources.

docs/modules/ROOT/pages/references/architecture/usage-profiles.adoc

@@ -0,0 +1,49 @@
+= Usage profiles
+
+Originally, the xref:appuio-cloud:ROOT:references/quality-requirements/performance/resource-quota.adoc[] and xref:appuio-cloud:ROOT:references/quality-requirements/performance/ns-quota.adoc[] requirements have been implemented as Kyverno policies on each zone.
+With this approach, overriding the default resource quotas must be done on individual namespaces (or on zones for the namespace count quota).
+This approach doesn't scale, and doesn't easily allow configuring different default quotas for different organizations or billing entities.
+
+As {product} has grown, and we now allow users to sign up autonomously (self signup), we want to configure lower default quotas and additional policies (for example network policies) for organizations of billing entities which haven't been validated yet.
+
+== Usage profiles
+
+We've decided to call a set of quotas and policies a _usage profile_.
+
+We're implementing usage policies in the {controlapi}, so that non-technical users who have administrative privileges can easily update the usage profile for a billing entity or an organization in the {portal}.
+
+Usage profiles are defined as a new custom resource in the {controlapi}.
+See the xref:appuio-cloud:ROOT:references/architecture/control-api-usage-profile.adoc[`UsageProfile` reference documentation] for details on the structure of the custom resource.
+
+=== Default usage profile for a billing entity
+
+We want to allow operators of {product} to set a default usage profile for a whole billing entity.
+To enable this, we introduce a new field `spec.usageProfileRef` on the billing entity which indicates the default usage profile for organizations associated with this billing entity.
+The {controlapi} sets field `status.usageProfileRef` of all organizations to field `spec.usageProfileRef` of the billing entity referenced in `spec.billingEntityRef`.
+
+=== Customize usage profile for an organization
+
+The organization virtual resource has a new optional field `spec.usageProfileRef` which contains the name (`metadata.name`) of a `UsageProfile` resource.
+If this field is present, it overrides the default usage profile that would be assigned to the organization based on its billing entity.
+
+If present, the value of this field is validated by the {controlapi}.
+The {controlapi} rejects updates of an organization virtual resource, if the `spec.usageProfileRef` field is modified and the principal doesn't have access to the usage profile referenced in the updated organization resource.
+
+The {controlapi} ensures that `status.usageProfileRef` is updated to the new billing entity's default usage profile when an organization's `spec.billingEntityRef` is updated, if the organization doesn't have `spec.usageProfileRef` set.
+
+== Applying usage profiles on zones
+
+To apply usage profiles to organization namespaces on each zone, we extend the {product} agent.
+The agent identifies the organization's usage profile by looking at field `status.usageProfileRef` in the {controlapi} `Organization` object for the organization.
+
+We reimplement the existing Kyverno policies which manages the resource quotas in organization namespaces and restrict the number of namespaces per zone in the agent.
+The resource quota policy is changed to apply the quotas and policies defined in the `UsageProfile` referenced by the `Organization` by default, instead of the default quotas stored on the zone.
+Per-namespace quota overrides are still possible through namespace annotations.
+Additionally, the agent reject new namespaces if the namespace's organization has reached its namespace count limit on the zone.
+The agent reads the organization's namespace count limit from the organization's usage profile.
+
+To determine the usage profile for each organization, the agent connects to the {controlapi} to read `UsageProfile` and `Organization` resources.
+The agent watches these resource for changes to ensure changes are reflected on the zones.
+
+Additionally, the agent reconciles all Kubernetes resources which are created to apply a usage profile.
+This ensures that users can't modify the quotas and policies defined through the usage profile.

docs/modules/ROOT/partials/nav.adoc

@@ -15,6 +15,8 @@
 ** xref:appuio-cloud:ROOT:references/architecture/metrics-of-interest.adoc[]
 ** xref:appuio-cloud:ROOT:references/architecture/metering-data-flow.adoc[]
 ** xref:appuio-cloud:ROOT:references/architecture/invitations.adoc[]
+** xref:appuio-cloud:ROOT:references/architecture/usage-profiles.adoc[]
+
 
 ** System
 *** xref:appuio-cloud:ROOT:explanation/system/context.adoc[]
@@ -30,6 +32,7 @@
 *** xref:appuio-cloud:ROOT:references/architecture/control-api-zone.adoc[Zone]
 *** xref:appuio-cloud:ROOT:references/architecture/control-api-billing-entity.adoc[BillingEntity]
 *** xref:appuio-cloud:ROOT:references/architecture/control-api-invitation.adoc[Invitation]
+*** xref:appuio-cloud:ROOT:references/architecture/control-api-usage-profile.adoc[UsageProfile]
 *** Adapters
 **** xref:appuio-cloud:ROOT:references/architecture/control-api-org-adapter.adoc[Organization Adapter]
 **** xref:appuio-cloud:ROOT:references/architecture/control-api-billing-entity-adapter.adoc[BillingEntity Adapter]