Skip to content

Latest commit

 

History

History
381 lines (324 loc) · 12.5 KB

kubearmor_visibility.md

File metadata and controls

381 lines (324 loc) · 12.5 KB
Raw

KubeArmor Visibility

KubeArmor currently supports enabling visibility for containers and hosts.

Visibility for hosts is not enabled by default, however it is enabled by default for containers .

The karmor tool provides access to both using karmor logs.

Available visibility options:

KubeArmor provides visibility on the following behavior of containers

  • Process
  • Files
  • Networks

Prerequisites

Example: wordpress-mysql

kubectl apply -f https://raw.githubusercontent.com/kubearmor/KubeArmor/main/examples/wordpress-mysql/security-policies/ksp-wordpress-block-process.yaml

This sample policy blocks execution of the apt and apt-get commands in wordpress pods with label selector app: wordpress.

Getting Container Visibility

  • Checking default visibility

    • Container visibility is enabled by default. We can check it using kubectl describe and grep kubearmor-visibility
    POD_NAME=$(kubectl get pods -n wordpress-mysql -l app=wordpress -o jsonpath='{.items[0].metadata.name}') && kubectl describe -n wordpress-mysql pod $POD_NAME | grep kubearmor-visibility

kubearmor-visibility: process, file, network, capabilities
    • For pre-existing workloads : Enable visibility using kubectl annotate. Currently KubeArmor supports process, file, network, capabilities
    kubectl annotate pods <pod-name> -n wordpress-mysql "kubearmor-visibility=process,file,network,capabilities"

  • Open up a terminal, and watch logs using the karmor cli

    karmor logs

  • In another terminal, simulate a policy violation . Try sleep inside a pod

    POD_NAME=$(kubectl get pods -n wordpress-mysql -l app=wordpress -o jsonpath='{.items[0].metadata.name}') && kubectl -n wordpress-mysql exec -it $POD_NAME -- bash
# apt update

  • In the terminal running karmor logs, the policy violation along with container visibility is shown, in this case for example

    Click to expand 
    == Alert / 2023-01-04 04:58:37.689182 ==
ClusterName: default
HostName: sibashi-asus
NamespaceName: wordpress-mysql
PodName: wordpress-787f45786f-mm2bm
Labels: app=wordpress
ContainerName: wordpress
ContainerID: 9af5246810fd0a732e74d391b32b95f65e4c08e655d1ab10b49b04b148cc1c24
ContainerImage: docker.io/library/wordpress:4.8-apache@sha256:6216f64ab88fc51d311e38c7f69ca3f9aaba621492b4f1fa93ddf63093768845
Type: MatchedPolicy
PolicyName: ksp-wordpress-block-process
Severity: 3
Source: /bin/bash
Resource: /usr/bin/apt update
Operation: Process
Action: Block
Data: syscall=SYS_EXECVE
Enforcer: AppArmor
Result: Permission denied
HostPID: 17462
HostPPID: 17293
PID: 199
PPID: 193
ParentProcessName: /bin/bash
ProcessName: /usr/bin/apt

  • The logs can also be generated in JSON format using karmor logs --json

    Click to expand 
    {
"Timestamp":1672808328,
"UpdatedTime":"2023-01-04T04:58:48.838991Z",
"ClusterName":"default","HostName":"sibashi-asus",
"NamespaceName":"wordpress-mysql","PodName":"wordpress-787f45786f-mm2bm",
"Labels":"app=wordpress",
"ContainerID":"9af5246810fd0a732e74d391b32b95f65e4c08e655d1ab10b49b04b148cc1c24",
"ContainerName":"wordpress",
"ContainerImage":"docker.io/library/wordpress:4.8-apache@sha256:6216f64ab88fc51d311e38c7f69ca3f9aaba621492b4f1fa93ddf63093768845",
"HostPPID":17293,
"HostPID":17526,
"PPID":193,
"PID":200,
"ParentProcessName":"/bin/bash",
"ProcessName":"/usr/bin/apt",
"PolicyName":"ksp-wordpress-block-process",
"Severity":"3",
"Type":"MatchedPolicy",
"Source":"/bin/bash",
"Operation":"Process",
"Resource":"/usr/bin/apt update",
"Data":"syscall=SYS_EXECVE",
"Enforcer":"AppArmor",
"Action":"Block",
"Result":"Permission denied"
}

Getting Host Visibility

  • Host Visibility is not enabled by default . To enable Host Visibility we need to annotate the node using kubectl annotate node
  kubectl annotate node <node-name> "kubearmor-visibility=process,file,network,capabilities"
  • To confirm it use kubectl describe and grep kubearmor-visibility
kubectl describe node <node-name> | grep kubearmor-visibility
  • Now we can get general telemetry events in the context of the host using karmor logs .The logs related to Host Visibility will have type Type: HostLogand Operation: File | Process | Network 
karmor logs --logFilter=all
Click to expand 
== Alert / 2023-01-04 04:58:37.689182 ==
== Log / 2023-01-27 14:41:49.017709 ==
ClusterName: default
HostName: kubearmor-dev2
Type: HostLog
Source: /usr/bin/dockerd
Resource: /usr/bin/runc --version
Operation: Process
Data: syscall=SYS_EXECVE
Result: Passed
HostPID: 193088
HostPPID: 914
PID: 193088
PPID: 914
ParentProcessName: /usr/bin/dockerd
ProcessName: /usr/bin/runc
== Log / 2023-01-27 14:41:49.018951 ==
ClusterName: default
HostName: kubearmor-dev2
Type: HostLog
Source: /usr/bin/runc --version
Resource: /lib/x86_64-linux-gnu/libc.so.6
Operation: File
Data: syscall=SYS_OPENAT fd=-100 flags=O_RDONLY|O_CLOEXEC
Result: Passed
HostPID: 193088
HostPPID: 914
PID: 193088
PPID: 914
ParentProcessName: /usr/bin/dockerd
ProcessName: /usr/bin/runc
== Log / 2023-01-27 14:41:49.018883 ==
ClusterName: default
HostName: kubearmor-dev2
Type: HostLog
Source: /usr/bin/runc --version
Resource: /etc/ld.so.cache
Operation: File
Data: syscall=SYS_OPENAT fd=-100 flags=O_RDONLY|O_CLOEXEC
Result: Passed
HostPID: 193088
HostPPID: 914
PID: 193088
PPID: 914
ParentProcessName: /usr/bin/dockerd
ProcessName: /usr/bin/runc
== Log / 2023-01-27 14:41:49.020905 ==
ClusterName: default
HostName: kubearmor-dev2
Type: HostLog
Source: /var/lib/rancher/k3s/data/2949af7261ce923f6a5091396d266a0e9d9436dcee976fcd548edc324eb277bb/bin/k3s
Resource: /var/lib/rancher/k3s/data/2949af7261ce923f6a5091396d266a0e9d9436dcee976fcd548edc324eb277bb/bin/portmap
Operation: Process
Data: syscall=SYS_EXECVE
Result: Passed
HostPID: 193090
HostPPID: 5627
PID: 193090
PPID: 5627
ParentProcessName: /var/lib/rancher/k3s/data/2949af7261ce923f6a5091396d266a0e9d9436dcee976fcd548edc324eb277bb/bin/k3s
ProcessName: /var/lib/rancher/k3s/data/2949af7261ce923f6a5091396d266a0e9d9436dcee976fcd548edc324eb277bb/bin/portmap

  • The logs can also be generated in JSON format using karmor logs --logFilter=all --json

    Click to expand 
        {
    "Timestamp": 1674889485,
    "UpdatedTime": "2023-01-28T07:04:45.436854Z",
    "ClusterName": "default",
    "HostName": "kubearmor-dev2",
    "ParentProcessName": "/usr/bin/containerd-shim",
    "ProcessName": "/usr/bin/containerd",
    "HostPPID": 80989,
    "HostPID": 614074,
    "PPID": 80989,
    "PID": 614074,
    "Type": "HostLog",
    "Source": "/usr/bin/containerd --address /run/containerd/containerd.sock publish --topic /tasks/exit --namespace moby",
    "Operation": "File",
    "Resource": "/lib/x86_64-linux-gnu/libdl.so.2",
    "Data": "syscall=SYS_OPENAT fd=-100 flags=O_RDONLY|O_CLOEXEC",
    "Result": "Passed",
  },
  {
  "Timestamp": 1674889485,
  "UpdatedTime": "2023-01-28T07:04:45.440804Z",
  "ClusterName": "default",
  "HostName": "kubearmor-dev2",
  "ParentProcessName": "/usr/bin/containerd-shim",
  "ProcessName": "/usr/bin/containerd",
  "HostPPID": 80989,
  "HostPID": 614074,
  "PPID": 80989,
  "PID": 614074,
  "Type": "HostLog",
  "Source": "/usr/bin/containerd --address /run/containerd/containerd.sock publish --topic /tasks/exit --namespace moby",
  "Operation": "File",
  "Resource": "/usr/bin/containerd",
  "Data": "syscall=SYS_OPENAT fd=-100 flags=O_RDONLY|O_CLOEXEC",
  "Result": "Passed",
  },
  {
  "Timestamp": 1674889485,
  "UpdatedTime": "2023-01-28T07:04:45.427658Z",
  "ClusterName": "default",
  "HostName": "kubearmor-dev2",
  "ParentProcessName": "/bin/bash",
  "ProcessName": "/usr/bin/pgrep",
  "HostPPID": 614066,
  "HostPID": 614073,
  "PPID": 614066,
  "PID": 614073,
  "Type": "HostLog",
  "Source": "/bin/bash",
  "Operation": "Process",
  "Resource": "/usr/bin/pgrep kubearmor",
  "Data": "syscall=SYS_EXECVE",
  "Result": "Passed",
  },
  {
  "Timestamp": 1674889485,
  "UpdatedTime": "2023-01-28T07:04:45.450454Z",
  "ClusterName": "default",
  "HostName": "kubearmor-dev2",
  "ParentProcessName": "/usr/bin/containerd-shim",
  "ProcessName": "/usr/bin/containerd",
  "HostPPID": 80989,
  "HostPID": 614074,
  "PPID": 80989,
  "PID": 614074,
  "Type": "HostLog",
  "Source": "/usr/bin/containerd --address /run/containerd/containerd.sock publish --topic /tasks/exit --namespace moby",
  "Operation": "Network",
  "Resource": "domain=AF_UNIX type=SOCK_STREAM|SOCK_NONBLOCK|SOCK_CLOEXEC protocol=0",
  "Data": "syscall=SYS_SOCKET",
  "Result": "Passed"
  }

Updating Namespace Visibility

KubeArmor has the ability to let the user select what kind of events have to be traced by changing the annotation kubearmor-visibility at the namespace.

  • Checking Namespace visibility

    • Namespace visibility can be checked using kubectl describe.
    kubectl describe ns wordpress-mysql | grep kubearmor-visibility

kubearmor-visibility: process, file, network, capabilities
    • To update the visibility of namespace : Now let's update Kubearmor visibility using kubectl annotate. Currently KubeArmor supports process, file, network, capabilities. Lets try to update visibility for the namespace wordpress-mysql
     kubectl annotate ns wordpress-mysql kubearmor-visibility=network --overwrite
 "namespace/wordpress-mysql annotated"

    Note: To turn off the visibility across all aspects, use kubearmor-visibility=none. Note that any policy violations or events that results in non-success returns would still be reported in the logs.

  • Open up a terminal, and watch logs using the karmor cli

    karmor logs --logFilter=all -n wordpress-mysql

  • In another terminal, let's exec into the pod and run some process commands . Try ls inside the pod

      POD_NAME=$(kubectl get pods -n wordpress-mysql -l app=wordpress -o jsonpath='{.items[0].metadata.name}') && kubectl -n wordpress-mysql exec -it $POD_NAME -- bash
# ls

    Now, we can notice that no logs have been generated for the above command and logs with only Operation: Network are shown.

    Note If telemetry is disabled, the user wont get audit event even if there is an audit rule.

    Note Only the logs are affected by changing the visibility, we still get all the alerts that are generated.

  • Let's simulate a sample policy violation, and see whether we still get alerts or not.

    • Policy violation :
    POD_NAME=$(kubectl get pods -n wordpress-mysql -l app=wordpress -o jsonpath='{.items[0].metadata.name}') && kubectl -n wordpress-mysql exec -it $POD_NAME -- bash
#apt

    Here, note that the alert with Operation: Process is reported.

    Click to expand 
    == Alert / 2023-04-21 10:54:16.167986 ==
ClusterName: default
HostName: aryan-vm
NamespaceName: wordpress-mysql
PodName: wordpress-c4bf5b44b-wsfkg
Labels: app=wordpress
ContainerName: wordpress
ContainerID: f6fa783eac62b3cc315059c349e88aa851bd87e3e8d4e91ac539dc2a6ca71ae6
ContainerImage: wordpress:4.8-apache@sha256:6216f64ab88fc51d311e38c7f69ca3f9aaba621492b4f1fa93ddf63093768845
Type: MatchedPolicy
PolicyName: ksp-wordpress-block-process
Severity: 3
Source: /bin/bash
Resource: /usr/bin/apt
Operation: Process
Action: Block
Data: syscall=SYS_EXECVE
Enforcer: AppArmor
Result: Permission denied
HostPID: 1.252628e+06
HostPPID: 1.251261e+06
PID: 200
PPID: 192
ParentProcessName: /bin/bash
ProcessName: /usr/bin/apt