New CS proposal: Kubernetes Security Cheatsheet #465

[ragashreeshekar]

Please refer to #465 for more details.

[mackowski]

@Seppl2202 @jmbmxer @c0d3xpl0it @ragashree-mc Please review this and add comments, suggestions etc.

[jmbmxer]

Hi Ragashree!

This looks really great! I have a few sections I'd like to propose below. These are not all 100% complete but figured I'd get them down first.

Service Mesh Security Policies (New)

Logging (Container Logs) (Expansion)

Managed Kubernetes Security Considerations (New)

Cluster Governance with Open Policy Agent (New)

Container Runtime Security (Additions)

Hardening containers at runtime gives security teams the ability to detect and respond to threats and anomalies while the containers or workloads are in a running state. This is typically carried out by intercepting the low-level system calls and looking for events that may indicate compromise. Some examples of events that should trigger an alert would include:

• A shell is run inside a container
• A container mounts a sensitive path from the host such as /proc
• A sensitive file is unexpectedly read in a running container such as /etc/shadow
• An outbound network connection is established

Open source tools such as Falco from Sysdig are available to help operators get up an running with container runtime security by providing a large number of out-of-the-box detections as well as the flexibility to create custom rules.

Container Sandboxing (New)

Container runtimes typically are permitted to make direct calls to the host kernel then the kernel interacts with hardware and devices to respond to the request. Cgroups and namespaces exist to give containers a certain amount of isolation but the still kernel presents a large attack surface area. Often times in multi-tenant and highly untrusted clusters an additional layer of sandboxing is required to ensure container breakout and kernel exploits are not present. Below we will explore a few OSS technologies that help further isolate running containers from the host kernel:

Kata Containers: Kata Containers is OSS project that uses stripped-down VMs to keep the resource footprint minimal and maximize performance to ultimately isolate containers further.

gVisor : gVisor is a more lightweight than a VM (even stripped down). gVisor is its own independent kernel written in Go to sit in the middle of a container and the host kernel. Strong sandbox. gVisor supports ~70% of the linux system calls from the container but ONLY users about 20 system calls to the host kernel.

Firecracker: Firecracker super lightweight VM that runs in user space. Locked down by seccomp, cgroup and namespace policies so system calls are very limited. Firecracker is built with security in mind but may not support all Kubernetes or container runtime deployments.

[Seppl2202]

This looks great. I wanted to mention Falco as a nice tool for monitoring, but jmbmxr was faster :)

I also wanted to ask of we should handle the Kubernetes Dashboard here and that it should run with its own account with least privileges, because Tesla was hacked through a dashboard that was not configured properly.

[c0d3xpl0it]

I think we should also add tools like Checkov for security scanning of IaC.

[ragashreeshekar]

Hi Ragashree!

This looks really great! I have a few sections I'd like to propose below. These are not all 100% complete but figured I'd get them down first.

Service Mesh Security Policies (New)

Logging (Container Logs) (Expansion)

Managed Kubernetes Security Considerations (New)

Cluster Governance with Open Policy Agent (New)

Container Runtime Security (Additions)

Hardening containers at runtime gives security teams the ability to detect and respond to threats and anomalies while the containers or workloads are in a running state. This is typically carried out by intercepting the low-level system calls and looking for events that may indicate compromise. Some examples of events that should trigger an alert would include:

• A shell is run inside a container
• A container mounts a sensitive path from the host such as /proc
• A sensitive file is unexpectedly read in a running container such as /etc/shadow
• An outbound network connection is established

Open source tools such as Falco from Sysdig are available to help operators get up an running with container runtime security by providing a large number of out-of-the-box detections as well as the flexibility to create custom rules.

Container Sandboxing (New)

Container runtimes typically are permitted to make direct calls to the host kernel then the kernel interacts with hardware and devices to respond to the request. Cgroups and namespaces exist to give containers a certain amount of isolation but the still kernel presents a large attack surface area. Often times in multi-tenant and highly untrusted clusters an additional layer of sandboxing is required to ensure container breakout and kernel exploits are not present. Below we will explore a few OSS technologies that help further isolate running containers from the host kernel:

Kata Containers: Kata Containers is OSS project that uses stripped-down VMs to keep the resource footprint minimal and maximize performance to ultimately isolate containers further.

gVisor : gVisor is a more lightweight than a VM (even stripped down). gVisor is its own independent kernel written in Go to sit in the middle of a container and the host kernel. Strong sandbox. gVisor supports ~70% of the linux system calls from the container but ONLY users about 20 system calls to the host kernel.

Firecracker: Firecracker super lightweight VM that runs in user space. Locked down by seccomp, cgroup and namespace policies so system calls are very limited. Firecracker is built with security in mind but may not support all Kubernetes or container runtime deployments.

Thanks a bunch, @jmbmxer. I will incorporate the additional sections in the documentation and I will also look into the suggested sections.

[Seppl2202]

So here is my proposal for the K8s Dashboard. I would insert it into the Securing Kubernetes components section:

The Kubernetes dashboard is a webapp for monitoring your cluster. It it is not a part of the Kubernetes cluster itself, it has to be installed by the owners of the cluster. Thus, there are a lot of tutorials on how to do this. Unfortunately, most of them create a service account with very high privileges. This caused Tesla and some others to be hacked via such a poorly configured K8s dashboard.

To prevent attacks via the dashboard, you should follow some tips:

• Do not expose the dashboard to the public. There is no need to access such a powerful tool from outside your LAN
• Turn on RBAC, so you can limit the service account the dashboard uses
• Do not grant the service account of the dashboard high privileges
• Grant permissions per user, so each user only can see what he is supposed to see
• If you are using network policies, you can block requests to the dashboard even from internal pods (this will not affect the proxy tunnel via kubectl proxy)
• Before version 1.8, the dashboard had a service account with full privileges, so check that there is no role binding for cluster-admin left.

Ressources:
https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/

https://blog.redlock.io/cryptojacking-tesla (not available anymore, that was the original report about the Tesla hack)

https://arstechnica.com/information-technology/2018/02/tesla-cloud-resources-are-hacked-to-run-cryptocurrency-mining-malware/

[ragashreeshekar]

So here is my proposal for the K8s Dashboard. I would insert it into the Securing Kubernetes components section:

The Kubernetes dashboard is a webapp for monitoring your cluster. It it is not a part of the Kubernetes cluster itself, it has to be installed by the owners of the cluster. Thus, there are a lot of tutorials on how to do this. Unfortunately, most of them create a service account with very high privileges. This caused Tesla and some others to be hacked via such a poorly configured K8s dashboard.

To prevent attacks via the dashboard, you should follow some tips:

• Do not expose the dashboard to the public. There is no need to access such a powerful tool from outside your LAN
• Turn on RBAC, so you can limit the service account the dashboard uses
• Do not grant the service account of the dashboard high privileges
• Grant permissions per user, so each user only can see what he is supposed to see
• If you are using network policies, you can block requests to the dashboard even from internal pods (this will not affect the proxy tunnel via kubectl proxy)
• Before version 1.8, the dashboard had a service account with full privileges, so check that there is no role binding for cluster-admin left.

Ressources:
https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/

https://blog.redlock.io/cryptojacking-tesla (not available anymore, that was the original report about the Tesla hack)

https://arstechnica.com/information-technology/2018/02/tesla-cloud-resources-are-hacked-to-run-cryptocurrency-mining-malware/

Thanks a bunch, @Seppl2202 . I have incorporated the suggested section :)

[ragashreeshekar]

I have updated the CS with a detailed logging session. Request help with the following sections:

Service Mesh Security Policies (New)
Managed Kubernetes Security Considerations (New)
Cluster Governance with Open Policy Agent (New)

I request @mackowski @Seppl2202 @c0d3xpl0it @jmbmxer and team your review and if the team is OK with the current content, we can request the repo owners to push it to drafts section as rightly suggested by @mackowski and we can work independently on enhancing the CS with the additonal content. Please let me know your thoughts.

[Seppl2202]

I have updated the CS with a detailed logging session. Request help with the following sections:

Service Mesh Security Policies (New)
Managed Kubernetes Security Considerations (New)
Cluster Governance with Open Policy Agent (New)

I request @mackowski @Seppl2202 @c0d3xpl0it @jmbmxer and team your review and if the team is OK with the current content, we can request the repo owners to push it to drafts section as rightly suggested by @mackowski and we can work independently on enhancing the CS with the additonal content. Please let me know your thoughts.

Looks great, if we can work independently, I could take the Service Mesh Security Policies (I think you mean Istio/LinkerD and maybe consul?)

[ragashreeshekar]

I have updated the CS with a detailed logging session. Request help with the following sections:

Service Mesh Security Policies (New)
Managed Kubernetes Security Considerations (New)
Cluster Governance with Open Policy Agent (New)

I request @mackowski @Seppl2202 @c0d3xpl0it @jmbmxer and team your review and if the team is OK with the current content, we can request the repo owners to push it to drafts section as rightly suggested by @mackowski and we can work independently on enhancing the CS with the additonal content. Please let me know your thoughts.

Looks great, if we can work independently, I could take the Service Mesh Security Policies (I think you mean Istio/LinkerD and maybe consul?)

Thank you! @Seppl2202
@mackowski Please let us know your thoughts.

[ragashreeshekar]

Gentle reminder, @mackowski @jmbmxer @jmanico, Please let us know the steps forward. Thanks in advance.

[mackowski]

LGTM

[mackowski]

@ragashreeshekar It looks good. I merged it to drafts and from now you can make PR to it independently untill we will be happy and ready to release it :-) Good job everyone!

[jmanico]

I'll be sure to carefully review this before the week is over, likely Thursday at the latest. Forgive me delay! Aloha, Jim

On 8/31/20 7:29 PM, Ragashree wrote: Gentle reminder, @mackowski <https://github.com/mackowski> @jmbmxer <https://github.com/jmbmxer> @jmanico <https://github.com/jmanico>, Please let us know the steps forward. Thanks in advance. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#470 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEBYCPRRCFHNPYALGDBYTLSDSBEFANCNFSM4QA5VDIQ>.

-- Jim Manico Manicode Security https://www.manicode.com