Add a webhook to reject the gatekeeper-ignore label on non-GK namespa…

…ces (#350) * Add a webhook to reject the gatekeeper-ignore label on non-GK namespaces Fixes #231 Signed-off-by: Max Smythe <smythe@google.com> * Namespace label webhook should fail hard Signed-off-by: Max Smythe <smythe@google.com> * Fix lint errors Signed-off-by: Max Smythe <smythe@google.com> * Wait for webhook on e2e tests; add gatekeeper label Signed-off-by: Max Smythe <smythe@google.com> * Remove unnecessary return code Signed-off-by: Max Smythe <smythe@google.com> * use --resolve instead of --connect-to for ubuntu compatibility Signed-off-by: Max Smythe <smythe@google.com> * Move cleaning of temp file closer to its creation Signed-off-by: Max Smythe <smythe@google.com> * Incorporate feedback from community mtg Signed-off-by: Max Smythe <smythe@google.com> * Fix manifests, add e2e tests Signed-off-by: Max Smythe <smythe@google.com> * Add README Signed-off-by: Max Smythe <smythe@google.com> * Add flag to helm chart Signed-off-by: Max Smythe <smythe@google.com> * Regenerate helm chart Signed-off-by: Max Smythe <smythe@google.com> * Update README to include `does not exlude audit` Signed-off-by: Max Smythe <smythe@google.com> * Add DR instructions to README Signed-off-by: Max Smythe <smythe@google.com> * Use staging manifests Signed-off-by: Max Smythe <smythe@google.com> * Add webhook customization readme and tweak flag name Signed-off-by: Max Smythe <smythe@google.com> * Add flag change to manifest Signed-off-by: Max Smythe <smythe@google.com> * Add remaining namespaces -> namespace flag changes Signed-off-by: Max Smythe <smythe@google.com> * Update deprecation comment Signed-off-by: Max Smythe <smythe@google.com>
open-policy-agent · Jan 24, 2020 · 5ebaa11 · 5ebaa11
1 parent a297913
commit 5ebaa11
Show file tree

Hide file tree

Showing 13 changed files with 497 additions and 16 deletions.
diff --git a/README.md b/README.md
@@ -358,6 +358,34 @@ status:
 ```
 > NOTE: The supported enforcementActions are [`deny`, `dryrun`] for constraints. Update the `--disable-enforcementaction-validation=true` flag if the desire is to disable enforcementAction validation against the list of supported enforcementActions.
 
+### Exempting Namespaces from the Gatekeeper Admission Webhook
+
+Note that the following only exempts resources from the admission webhook. They will still be audited. Editing individual constraints is
+necessary to exclude them from audit.
+
+If it becomes necessary to exempt a namespace from Gatekeeper entirely (e.g. you want `kube-system` to bypass admission checks), here's how to do it:
+
+   1. Make sure the validating admission webhook configuration for Gatekeeper has the following namespace selector:
+
+        ```yaml
+          namespaceSelector:
+            matchExpressions:
+            - key: admission.gatekeeper.sh/ignore
+              operator: DoesNotExist
+        ```
+      the default Gatekeeper manifest should already have added this. The default name for the
+      webhook configuration is `gatekeeper-validating-webhook-configuration` and the default
+      name for the webhook that needs the namespace selector is `validation.gatekeeper.sh`
+
+   2. Tell Gatekeeper it's okay for the namespace to be ignored by adding a flag to the pod:
+      `--exempt-namespace=<NAMESPACE NAME>`. This step is necessary because otherwise the
+      permission to modify a namespace would be equivalent to the permission to exempt everything
+      in that namespace from policy checks. This way a user must explicitly have permissions
+      to configure the Gatekeeper pod before they can add exemptions.
+
+   3. Add the `admission.gatekeeper.sh/ignore` label to the namespace. The value attached
+      to the label is ignored, so it can be used to annotate the reason for the exemption.
+
 ### Debugging
 
 > NOTE: Verbose logging with DEBUG level can be turned on with `--log-level=DEBUG`.  By default, the `--log-level` flag is set to minimum log level `INFO`. Acceptable values for minimum log level are [`DEBUG`, `INFO`, `WARNING`, `ERROR`]. In production, this flag should not be set to `DEBUG`.
@@ -411,6 +439,55 @@ When applying the constraint using `kubectl apply -f constraint.yaml` with a Con
 
 To find the error, run `kubectl get -f [CONSTRAINT_FILENAME].yaml -oyaml`. Build errors are shown in the `status` field.
 
+### Customizing Admission Behavior
+
+Gatekeeper is a [Kubernetes admission webhook](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#webhook-configuration)
+whose default configuration can be found in the `gatekeeper.yaml` manifest file. By default, it is
+a `ValidatingWebhookConfiguration` resource named `gatekeeper-validating-webhook-configuration`.
+
+Currently the configuration specifies two webhooks: one for checking a request against
+the installed constraints and a second webhook for checking labels on namespace requests
+that would result in bypassing constraints for the namespace. The namespace-label webhook
+is necessary to prevent a privilege escalation where the permission to add a label to a
+namespace is equivalent to the ability to bypass all constraints for that namespace.
+You can read more about the ability to exempt namespaces by label [above](#exempting-namespaces-from-the-gatekeeper-admission-webhook).
+
+Because Kubernetes adds features with each version, if you want to know how the webhook can be configured it
+is best to look at the official documentation linked at the top of this section. However, two particularly important
+configuration options deserve special mention: [timeouts](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#timeouts) and
+[failure policy](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#failure-policy).
+
+Timeouts allow you to configure how long the API server will wait for a response from the admission webhook before it
+considers the request to have failed. Note that setting the timeout longer than the overall request timeout
+means that the main request will time out before the webhook's failure policy is invoked.
+
+Failure policy controls what happens when a webhook fails for whatever reason. Common
+failure scenarios include timeouts, a 5xx error from the server or the webhook being unavailable.
+You have the option to ignore errors, allowing the request through, or failing, rejecting the request.
+This results in a direct tradeoff between availability and enforcement.
+
+Currently Gatekeeper is defaulting to using `Ignore` for the constraint requests. This is because
+the webhook server currently only has one instance, which risks downtime during actions like upgrades.
+As the theoretical availability improves we will likely change the default to `Fail`.
+
+The namespace label webhook defaults to `Fail`, this is to help ensure that policies preventing
+labels that bypass the webhook from being applied are enforced. Because this webhook only gets
+called for namespace modification requests, the impact of downtime is mitigated, making the
+theoretical maximum availability less of an issue.
+
+Because the manifest is available for customization, the webhook configuration can
+be tuned to meet your specific needs if they differ from the defaults.
+
+### Emergency Recovery
+
+If a situation arises where Gatekeeper is preventing the cluster from operating correctly,
+the webhook can be disabled. This will remove all Gatekeeper admission checks. Assuming
+the default webhook name has been used this can be achieved by running:
+
+`kubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io gatekeeper-validating-webhook-configuration`
+
+Redeploying the webhook configuration will re-enable Gatekeeper.
+
 ## Kick The Tires
 
 The [demo/basic](https://github.com/open-policy-agent/gatekeeper/tree/master/demo/basic) directory contains the above examples of simple constraints, templates and configs to play with. The [demo/agilebank](https://github.com/open-policy-agent/gatekeeper/tree/master/demo/agilebank) directory contains more complex examples based on a slightly more realistic scenario. Both folders have a handy demo script to step you through the demos.
@@ -419,17 +496,6 @@ The [demo/basic](https://github.com/open-policy-agent/gatekeeper/tree/master/dem
 
 ## Finalizers
 
-### Why does Gatekeeper add sync finalizers?
-
-When Gatekeeper syncs resources it's adding them to OPA's internal cache. This
-cache may be used by constraints to render decisions. Because of this stale data
-is bad. It can lead to invalid rejections (e.g. when a uniqueness constraint is
-violated because an update conflicts with a since-deleted resource), or invalid
-acceptance (e.g. if a constraint uses the cache to make sure a Deployment exists
-before a Service can be created). Finalizers help avoid stale state by making
-sure Gatekeeper has processed the deletion and removed the object from its cache
-before the API Server can garbage collect the object.
-
 ### How can I remove finalizers? Why are they hanging around?
 
 If Gatekeeper is running, it should automatically clean up the finalizer. If it
@@ -448,7 +514,4 @@ If Gatekeeper is not running:
   * The container was sent a hard kill signal
   * The container had a panic
 
-It is safest to remove the Config resource before uninstalling Gatekeeper, as
-that causes finalizers to be removed outside of the normal GC process.
-
 Finalizers can be removed manually via `kubectl edit` or `kubectl patch`
diff --git a/config/default/kustomization.yaml b/config/default/kustomization.yaml
@@ -9,8 +9,8 @@ namespace: gatekeeper-system
 namePrefix: gatekeeper-
 
 # Labels to add to all resources and selectors.
-#commonLabels:
-#  someName: someValue
+commonLabels:
+  gatekeeper.sh/system: "yes"
 
 bases:
 - ../crd

diff --git a/config/manager/manager.yaml b/config/manager/manager.yaml
@@ -3,6 +3,7 @@ kind: Namespace
 metadata:
   labels:
     control-plane: controller-manager
+    admission.gatekeeper.sh/ignore: no-self-managing
   name: system
 ---
 apiVersion: apps/v1
@@ -30,6 +31,7 @@ spec:
           - "--audit-interval=30"
           - "--port=8443"
           - "--logtostderr"
+          - "--exempt-namespace=gatekeeper-system"
         image: quay.io/open-policy-agent/gatekeeper:v3.1.0-beta.4
         imagePullPolicy: Always
         name: manager

diff --git a/config/webhook/manifests.yaml b/config/webhook/manifests.yaml
@@ -6,6 +6,24 @@ metadata:
   creationTimestamp: null
   name: validating-webhook-configuration
 webhooks:
+- clientConfig:
+    caBundle: Cg==
+    service:
+      name: webhook-service
+      namespace: system
+      path: /v1/admitlabel
+  failurePolicy: Fail
+  name: check-ignore-label.gatekeeper.sh
+  rules:
+  - apiGroups:
+    - ""
+    apiVersions:
+    - '*'
+    operations:
+    - CREATE
+    - UPDATE
+    resources:
+    - namespaces
 - clientConfig:
     caBundle: Cg==
     service:

diff --git a/config/webhook/webhook_patch.yaml b/config/webhook/webhook_patch.yaml
@@ -16,5 +16,12 @@ webhooks:
     timeoutSeconds: 5
     namespaceSelector:
       matchExpressions:
+        # using the control-plane label to bypass Gatekeeper is deprecated
+        # and will be removed from the default config in a future version
         - key: control-plane
           operator: DoesNotExist
+        - key: admission.gatekeeper.sh/ignore
+          operator: DoesNotExist
+  - name: check-ignore-label.gatekeeper.sh
+    sideEffects: None
+    timeoutSeconds: 5
diff --git a/manifest_staging/chart/gatekeeper-operator/helm-modifications/helm-modifications.yaml b/manifest_staging/chart/gatekeeper-operator/helm-modifications/helm-modifications.yaml
@@ -54,6 +54,7 @@ spec:
             - --logtostderr
             - --constraint-violations-limit={{ .Values.constraintViolationsLimit }}
             - --audit-from-cache={{ .Values.auditFromCache }}
+            - --exempt-namespace=gatekeeper-system
           imagePullPolicy: "{{ .Values.image.pullPolicy }}"
           image: "{{ .Values.image.repository }}:{{ .Values.image.release }}"
           resources: HELMSUBST_DEPLOYMENT_CONTAINER_RESOURCES

diff --git a/manifest_staging/chart/gatekeeper-operator/templates/gatekeeper.yaml b/manifest_staging/chart/gatekeeper-operator/templates/gatekeeper.yaml
@@ -2,9 +2,11 @@ apiVersion: v1
 kind: Namespace
 metadata:
   labels:
+    admission.gatekeeper.sh/ignore: no-self-managing
     app: '{{ template "gatekeeper-operator.name" . }}'
     chart: '{{ template "gatekeeper-operator.name" . }}'
     control-plane: controller-manager
+    gatekeeper.sh/system: "yes"
     heritage: '{{ .Release.Service }}'
     release: '{{ .Release.Name }}'
   name: gatekeeper-system
@@ -20,6 +22,7 @@ metadata:
   labels:
     app: '{{ template "gatekeeper-operator.name" . }}'
     chart: '{{ template "gatekeeper-operator.name" . }}'
+    gatekeeper.sh/system: "yes"
     heritage: '{{ .Release.Service }}'
     release: '{{ .Release.Name }}'
   name: configs.config.gatekeeper.sh
@@ -220,6 +223,7 @@ metadata:
   labels:
     app: '{{ template "gatekeeper-operator.name" . }}'
     chart: '{{ template "gatekeeper-operator.name" . }}'
+    gatekeeper.sh/system: "yes"
     heritage: '{{ .Release.Service }}'
     release: '{{ .Release.Name }}'
   name: gatekeeper-admin
@@ -232,6 +236,7 @@ metadata:
   labels:
     app: '{{ template "gatekeeper-operator.name" . }}'
     chart: '{{ template "gatekeeper-operator.name" . }}'
+    gatekeeper.sh/system: "yes"
     heritage: '{{ .Release.Service }}'
     release: '{{ .Release.Name }}'
   name: gatekeeper-manager-role
@@ -257,6 +262,7 @@ metadata:
   labels:
     app: '{{ template "gatekeeper-operator.name" . }}'
     chart: '{{ template "gatekeeper-operator.name" . }}'
+    gatekeeper.sh/system: "yes"
     heritage: '{{ .Release.Service }}'
     release: '{{ .Release.Name }}'
   name: gatekeeper-manager-role
@@ -352,6 +358,7 @@ metadata:
   labels:
     app: '{{ template "gatekeeper-operator.name" . }}'
     chart: '{{ template "gatekeeper-operator.name" . }}'
+    gatekeeper.sh/system: "yes"
     heritage: '{{ .Release.Service }}'
     release: '{{ .Release.Name }}'
   name: gatekeeper-manager-rolebinding
@@ -371,6 +378,7 @@ metadata:
   labels:
     app: '{{ template "gatekeeper-operator.name" . }}'
     chart: '{{ template "gatekeeper-operator.name" . }}'
+    gatekeeper.sh/system: "yes"
     heritage: '{{ .Release.Service }}'
     release: '{{ .Release.Name }}'
   name: gatekeeper-manager-rolebinding
@@ -389,6 +397,7 @@ metadata:
   labels:
     app: '{{ template "gatekeeper-operator.name" . }}'
     chart: '{{ template "gatekeeper-operator.name" . }}'
+    gatekeeper.sh/system: "yes"
     heritage: '{{ .Release.Service }}'
     release: '{{ .Release.Name }}'
   name: gatekeeper-webhook-server-cert
@@ -400,6 +409,7 @@ metadata:
   labels:
     app: '{{ template "gatekeeper-operator.name" . }}'
     chart: '{{ template "gatekeeper-operator.name" . }}'
+    gatekeeper.sh/system: "yes"
     heritage: '{{ .Release.Service }}'
     release: '{{ .Release.Name }}'
   name: gatekeeper-webhook-service
@@ -412,6 +422,7 @@ spec:
     app: '{{ template "gatekeeper-operator.name" . }}'
     chart: '{{ template "gatekeeper-operator.name" . }}'
     control-plane: controller-manager
+    gatekeeper.sh/system: "yes"
     heritage: '{{ .Release.Service }}'
     release: '{{ .Release.Name }}'
 ---
@@ -422,6 +433,7 @@ metadata:
     app: '{{ template "gatekeeper-operator.name" . }}'
     chart: '{{ template "gatekeeper-operator.name" . }}'
     control-plane: controller-manager
+    gatekeeper.sh/system: "yes"
     heritage: '{{ .Release.Service }}'
     release: '{{ .Release.Name }}'
   name: gatekeeper-controller-manager
@@ -433,6 +445,7 @@ spec:
       app: '{{ template "gatekeeper-operator.name" . }}'
       chart: '{{ template "gatekeeper-operator.name" . }}'
       control-plane: controller-manager
+      gatekeeper.sh/system: "yes"
       heritage: '{{ .Release.Service }}'
       release: '{{ .Release.Name }}'
   template:
@@ -441,6 +454,7 @@ spec:
         app: '{{ template "gatekeeper-operator.name" . }}'
         chart: '{{ template "gatekeeper-operator.name" . }}'
         control-plane: controller-manager
+        gatekeeper.sh/system: "yes"
         heritage: '{{ .Release.Service }}'
         release: '{{ .Release.Name }}'
     spec:
@@ -451,6 +465,7 @@ spec:
         - --logtostderr
         - --constraint-violations-limit={{ .Values.constraintViolationsLimit }}
         - --audit-from-cache={{ .Values.auditFromCache }}
+        - --exempt-namespace=gatekeeper-system
         command:
         - /manager
         env:
@@ -516,6 +531,7 @@ metadata:
   labels:
     app: '{{ template "gatekeeper-operator.name" . }}'
     chart: '{{ template "gatekeeper-operator.name" . }}'
+    gatekeeper.sh/system: "yes"
     heritage: '{{ .Release.Service }}'
     release: '{{ .Release.Name }}'
   name: gatekeeper-validating-webhook-configuration
@@ -532,6 +548,8 @@ webhooks:
     matchExpressions:
     - key: control-plane
       operator: DoesNotExist
+    - key: admission.gatekeeper.sh/ignore
+      operator: DoesNotExist
   rules:
   - apiGroups:
     - '*'
@@ -544,3 +562,23 @@ webhooks:
     - '*'
   sideEffects: None
   timeoutSeconds: 5
+- clientConfig:
+    caBundle: Cg==
+    service:
+      name: gatekeeper-webhook-service
+      namespace: gatekeeper-system
+      path: /v1/admitlabel
+  failurePolicy: Fail
+  name: check-ignore-label.gatekeeper.sh
+  rules:
+  - apiGroups:
+    - ""
+    apiVersions:
+    - '*'
+    operations:
+    - CREATE
+    - UPDATE
+    resources:
+    - namespaces
+  sideEffects: None
+  timeoutSeconds: 5