Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: Policy Audit Mode improvements #23591

Merged
merged 1 commit into from
Feb 8, 2023
Merged

docs: Policy Audit Mode improvements #23591

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion Documentation/security/gsg_sw_demo.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ Each pod will be represented in Cilium as an :ref:`endpoint`. We can invoke the
NAME READY STATUS RESTARTS AGE
cilium-5ngzd 1/1 Running 0 3m19s

$ kubectl -n kube-system exec cilium-1c2cz -- cilium endpoint list
$ kubectl -n kube-system exec ds/cilium -- cilium endpoint list
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
232 Disabled Disabled 16530 k8s:class=deathstar 10.0.0.147 ready
Expand Down
179 changes: 105 additions & 74 deletions Documentation/security/policy-creation.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,21 +11,33 @@ Creating Policies from Verdicts
*******************************

Policy Audit Mode configures Cilium to allow all traffic while logging all
connections that would otherwise be dropped by policy. Policy Audit Mode may be
configured for the entire daemon using ``--policy-audit-mode=true`` or for
individual Cilium Endpoints. When Policy Audit Mode is enabled, no network
policy is enforced so this setting is **not recommended for production
connections that would otherwise be dropped by network policies. Policy Audit
Mode may be configured for the entire daemon using ``--policy-audit-mode=true``
or for individual Cilium Endpoints. When Policy Audit Mode is enabled, no
network policy is enforced so this setting is **not recommended for production
deployment**. Policy Audit Mode supports auditing network policies implemented
at networks layers 3 and 4. This guide walks through the process of creating
policies using Policy Audit Mode.

.. include:: gsg_requirements.rst
.. include:: gsg_sw_demo.rst

Scale down the deathstar Deployment
===================================

In this guide we're going to scale down the deathstar Deployment in order to
next steps:

.. code-block:: shell-session

$ kubectl scale --replicas=1 deployment deathstar
deployment.apps/deathstar scaled

Enable Policy Audit Mode (Entire Daemon)
========================================

To observe policy audit messages for all endpoints managed by this daemonset, modify the Cilium configmap and restart all daemons:
To observe policy audit messages for all endpoints managed by this Daemonset,
modify the Cilium ConfigMap and restart all daemons:

.. tabs::

Expand Down Expand Up @@ -69,19 +81,27 @@ the ``cilium`` tool on the endpoint's Kubernetes node. The steps include:

#. Determine the endpoint id on which Policy Audit Mode will be enabled.
#. Identify the Cilium pod running on the same Kubernetes node corresponding to the endpoint.
#. Using the Cilium pod above, modify the endpoint config by setting ``PolicyAuditMode=Enabled``.
#. Using the Cilium pod above, modify the endpoint configuration by setting ``PolicyAuditMode=Enabled``.

The following shell commands perform these steps:

.. code-block:: shell-session

$ export PODNAME=deathstar
$ export NODENAME=$(kubectl get pod -o jsonpath="{.items[?(@.metadata.name=='$PODNAME')].spec.nodeName}")
$ export ENDPOINT=$(kubectl get cep -o jsonpath="{.items[?(@.metadata.name=='$PODNAME')].status.id}")
$ export CILIUM_POD=$(kubectl -n "$CILIUM_NAMESPACE" get pod --all-namespaces --field-selector spec.nodeName="$NODENAME" -lk8s-app=cilium -o jsonpath='{.items[*].metadata.name}')
$ kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- cilium endpoint config "$ENDPOINT" PolicyAuditMode=Enabled
Endpoint 232 configuration updated successfully
$ PODNAME=$(kubectl get pods -l app.kubernetes.io/name=deathstar -o jsonpath='{.items[*].metadata.name}')
$ NODENAME=$(kubectl get pod -o jsonpath="{.items[?(@.metadata.name=='$PODNAME')].spec.nodeName}")
$ ENDPOINT=$(kubectl get cep -o jsonpath="{.items[?(@.metadata.name=='$PODNAME')].status.id}")
$ CILIUM_POD=$(kubectl -n "$CILIUM_NAMESPACE" get pod --all-namespaces --field-selector spec.nodeName="$NODENAME" -lk8s-app=cilium -o jsonpath='{.items[*].metadata.name}')
$ kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- \
cilium endpoint config "$ENDPOINT" PolicyAuditMode=Enabled
Endpoint 232 configuration updated successfully

We can check that Policy Audit Mode is enabled for this endpoint with

.. code-block:: shell-session

$ kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- \
cilium endpoint get "$ENDPOINT" -o jsonpath='{[*].spec.options.PolicyAuditMode}'
Enabled

.. _observe_policy_verdicts:

Expand All @@ -90,9 +110,9 @@ Observe policy verdicts

In this example, we are tasked with applying security policy for the deathstar.
First, from the Cilium pod we need to monitor the notifications for policy
verdicts using ``cilium monitor -t policy-verdict``. We'll be monitoring for
inbound traffic towards the deathstar to identify that traffic and determine
whether to extend the network policy to allow that traffic.
verdicts using the Hubble CLI. We'll be monitoring for inbound traffic towards
the deathstar to identify it and determine whether to extend the network policy
to allow that traffic.

Apply a default-deny policy:

Expand All @@ -112,61 +132,47 @@ To apply this policy, run:
$ kubectl create -f \ |SCM_WEB|\/examples/minikube/sw_deny_policy.yaml
ciliumnetworkpolicy.cilium.io/empire-default-deny created

With the above policy, we will enable default-deny posture on ingress to
pods with the label ``org=empire`` and enable the policy verdict
notifications for those pods. The same principle applies on egress as well.
With the above policy, we will enable a default-deny posture on ingress to pods
with the label ``org=empire`` and enable the policy verdict notifications for
those pods. The same principle applies on egress as well.

From another terminal with kubectl access, send some traffic from the
tiefighter to the deathstar:
Now let's send some traffic from the tiefighter to the deathstar:

.. code-block:: shell-session

$ kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landed

Back in the Cilium pod, the policy verdict logs are printed in the monitor
output:
We can check the policy verdict from the Cilium Pod:

.. code-block:: shell-session

# cilium monitor -t policy-verdict
...
Policy verdict log: flow 0x63113709 local EP ID 232, remote ID 31028, proto 6, ingress, action audit, match none, 10.0.0.112 :54134 -> 10.29.50.40:80 tcp SYN

In the above example, we can see that endpoint ``232`` has received traffic
(``ingress true``) which doesn't match the policy (``action audit match
none``). The source of this traffic has the identity ``31028``. Let's gather a
bit more information about what these numbers mean:
$ kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- \
hubble observe flows -t policy-verdict --last 1
Feb 7 12:53:39.168: default/tiefighter:54134 (ID:31028) -> default/deathstar-6fb5694d48-5hmds:80 (ID:16530) policy-verdict:none AUDITED (TCP Flags: SYN)

.. code-block:: shell-session

# cilium endpoint list
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
232 Disabled (Audit) Disabled 16530 k8s:class=deathstar 10.29.50.40 ready
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=default
k8s:org=empire
...
# cilium identity get 31028
ID LABELS
31028 k8s:class=tiefighter
k8s:io.cilium.k8s.policy.cluster=default
k8s:io.cilium.k8s.policy.serviceaccount=default
k8s:io.kubernetes.pod.namespace=default
k8s:org=empire
In the above example, we can see that the Pod ``deathstar-6fb5694d48-5hmds`` has
received traffic from the ``tiefighter`` Pod which doesn't match the policy
(``policy-verdict:none AUDITED``).

.. _create_network_policy:

Create the Network Policy
=========================

Given the above information, we now know the labels of the target pod, the
labels of the peer that's attempting to connect, the direction of the traffic
and the port. In this case, we can see clearly that it's an empire craft
so once we've determined that we expect this traffic to arrive at the
deathstar, we can form a policy to match the traffic:
We can get more information about the flow with

.. code-block:: shell-session

$ kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- \
hubble observe flows -t policy-verdict -o json --last 1

Given the above information, we now know the labels of the source and
destination Pods, the traffic direction, and the destination port. In this case,
we can see clearly that the source (i.e. the tiefighter Pod) is an empire
aircraft (as it has the ``org=empire`` label) so once we've determined that we
expect this traffic to arrive at the deathstar, we can form a policy to match
the traffic:

.. literalinclude:: ../../examples/minikube/sw_l3_l4_policy.yaml

Expand All @@ -177,24 +183,25 @@ To apply this L3/L4 policy, run:
$ kubectl create -f \ |SCM_WEB|\/examples/minikube/sw_l3_l4_policy.yaml
ciliumnetworkpolicy.cilium.io/rule1 created

Now if we run the landing requests again, we can observe in the monitor output
that the traffic which was previously audited to be dropped by the policy are
now reported as allowed:
Now if we run the landing requests again,

.. code-block:: shell-session

$ kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landed

Executed from the cilium pod:
we can then observe that the traffic which was previously audited to be dropped
by the policy is reported as allowed:

.. code-block:: shell-session

# cilium monitor -t policy-verdict
Policy verdict log: flow 0xabf3bda6 local EP ID 232, remote ID 31028, proto 6, ingress, action allow, match L3-L4, 10.0.0.112 :59824 -> 10.0.0.147:80 tcp SYN
$ kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- \
hubble observe flows -t policy-verdict --last 1
...
Feb 7 13:06:45.130: default/tiefighter:59824 (ID:31028) -> default/deathstar-6fb5694d48-5hmds:80 (ID:16530) policy-verdict:L3-L4 ALLOWED (TCP Flags: SYN)

Now the policy verdict states that the traffic would be allowed: ``action
allow``. Success!
Now the policy verdict states that the traffic would be allowed:
``policy-verdict:L3-L4 ALLOWED``. Success!

Disable Policy Audit Mode (Entire Daemon)
=========================================
Expand Down Expand Up @@ -234,35 +241,59 @@ These steps are nearly identical to enabling Policy Audit Mode.

.. code-block:: shell-session

$ export PODNAME=deathstar
$ export NODENAME=$(kubectl get pod -o jsonpath="{.items[?(@.metadata.name=='$PODNAME')].spec.nodeName}")
$ export ENDPOINT=$(kubectl get cep -o jsonpath="{.items[?(@.metadata.name=='$PODNAME')].status.id}")
$ export CILIUM_POD=$(kubectl -n "$CILIUM_NAMESPACE" get pod --all-namespaces --field-selector spec.nodeName="$NODENAME" -lk8s-app=cilium -o jsonpath='{.items[*].metadata.name}')
$ kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- cilium endpoint config "$ENDPOINT" PolicyAuditMode=Disabled
Endpoint 232 configuration updated successfully
$ PODNAME=$(kubectl get pods -l app.kubernetes.io/name=deathstar -o jsonpath='{.items[*].metadata.name}')
$ NODENAME=$(kubectl get pod -o jsonpath="{.items[?(@.metadata.name=='$PODNAME')].spec.nodeName}")
$ ENDPOINT=$(kubectl get cep -o jsonpath="{.items[?(@.metadata.name=='$PODNAME')].status.id}")
$ CILIUM_POD=$(kubectl -n "$CILIUM_NAMESPACE" get pod --all-namespaces --field-selector spec.nodeName="$NODENAME" -lk8s-app=cilium -o jsonpath='{.items[*].metadata.name}')
$ kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- \
cilium endpoint config "$ENDPOINT" PolicyAuditMode=Disabled
Endpoint 232 configuration updated successfully

Alternatively, **restarting the Cilium pod** will set the endpoint Policy Audit Mode to the daemon set configuration.


Verify Policy Audit Mode Is Disabled
Verify Policy Audit Mode is Disabled
====================================

.. code-block:: shell-session

$ kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- \
cilium endpoint get "$ENDPOINT" -o jsonpath='{[*].spec.options.PolicyAuditMode}'
Disabled

Now if we run the landing requests again, only the *tiefighter* pods with the
label ``org=empire`` will succeed. The *xwing* pods will be blocked!
label ``org=empire`` should succeed:

.. code-block:: shell-session

$ kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landed

This works as expected. Now the same request run from an *xwing* pod will fail:
And we can observe that the traffic was allowed by the policy:

.. code-block:: shell-session

$ kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- \
hubble observe flows -t policy-verdict --from-pod tiefighter --last 1
Feb 7 13:34:26.112: default/tiefighter:37314 (ID:31028) -> default/deathstar-6fb5694d48-5hmds:80 (ID:16530) policy-verdict:L3-L4 ALLOWED (TCP Flags: SYN)


This works as expected. Now the same request from an *xwing* Pod should fail:

.. code-block:: shell-session

$ kubectl exec xwing -- curl --connect-timeout 3 -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
command terminated with exit code 28

This curl request should timeout after three seconds, we can observe the policy
verdict with:

.. code-block:: shell-session

$ kubectl exec xwing -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
$ kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- \
hubble observe flows -t policy-verdict --from-pod xwing --last 1
Feb 7 13:43:46.791: default/xwing:54842 (ID:22654) <> default/deathstar-6fb5694d48-5hmds:80 (ID:16530) policy-verdict:none DENIED (TCP Flags: SYN)

This request will hang, so press Control-C to kill the curl request, or wait
for it to time out.

We hope you enjoyed the tutorial. Feel free to play more with the setup,
follow the `gs_http` guide, and reach out to us on the `Cilium
Expand All @@ -274,4 +305,4 @@ Clean-up
.. parsed-literal::

$ kubectl delete -f \ |SCM_WEB|\/examples/minikube/http-sw-app.yaml
$ kubectl delete cnp rule1
$ kubectl delete cnp empire-default-deny rule1