Not able to connect to Firecracker VM from the pod

[anubhavjana]

I have a deployment file which is meant to boot a firecracker VM on the pod. I have installed python and pip inside the ex4 filesystem. I want to execute the python script from the host / pod (either through CLI or inside yaml specification). The pod starts running, the curl requests are successful.

Here is a sample deployment file i am using:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cnn-fc
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cnn-fc
  template:
    metadata:
      labels:
        app: cnn-fc
    spec:    
      hostPID: true  # Required for accessing /dev/kvm
      containers:
        - name: cnn-fc
          image: localhost:5000/ubuntu22.04-updated:latest
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true  # Firecracker needs KVM access          
          command: ["/bin/bash", "-c"]
          args:
            - |
              echo "-------- Starting Firecracker VM --------"
              SOCKET_PATH="/run/firecracker-${POD_NAME}.sock"
              rm -f $SOCKET_PATH  # Ensure no stale socket
              /usr/local/bin/firecracker --api-sock $SOCKET_PATH > /var/lib/firecracker.log 2>&1 &
              FC_PID=$!
              echo "Firecracker started with PID: $FC_PID on socket: $SOCKET_PATH"              

              # echo "-------- Checking Firecracker process --------"
              # ps aux | grep firecracker

              # Configure Firecracker VM
              echo "------- Configuring Firecracker boot source --------"
              curl --unix-socket $SOCKET_PATH -X PUT "http://localhost/boot-source" \
              -H "Content-Type: application/json" \
              -d '{
                      "kernel_image_path": "/var/lib/firecracker-containerd/runtime/hello-vmlinux.bin",
                      "boot_args": "console=ttyS0 reboot=k panic=1 pci=off selinux=0 quiet loglevel=0"
                  }' 

              

              # Attach root filesystem
              echo "-------- Attaching root filesystem --------"
              curl --unix-socket $SOCKET_PATH -X PUT "http://localhost/drives/rootfs" \
              -H "Content-Type: application/json" \
              -d '{
                      "drive_id": "rootfs",
                      "path_on_host": "/var/lib/firecracker-containerd/runtime/ubuntu-24.04.ext4",
                      "is_root_device": true,
                      "is_read_only": true
                  }'             

              echo "Root filesystem attached successfully!"             

              # Start VM
              echo "-------- Starting Firecracker VM --------"
              curl --unix-socket $SOCKET_PATH -X PUT "http://localhost/actions" \
              -H "Content-Type: application/json" \
              -d '{
                      "action_type": "InstanceStart"
                  }'
             
              echo "Firecracker VM started successfully!"

              tail -f /dev/null

          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name  # Unique socket per pod
          volumeMounts:
            - name: firecracker-socket
              mountPath: /run
            - name: firecracker-binary
              mountPath: /usr/local/bin/firecracker  
            - name: firecracker-images
              mountPath: /var/lib/firecracker-containerd/runtime  
            
      volumes:
        - name: firecracker-socket
          hostPath:
            path: /run
            type: Directory
        - name: firecracker-binary
          hostPath:
            path: /usr/local/bin/firecracker  # Firecracker binary on host
            type: File
        - name: firecracker-images
          hostPath:
            path: /var/lib/firecracker-containerd/runtime  # Kernel & RootFS images
            type: Directory

Please let me know what other information you need from my side. Please help.

[Manciukic]

Hey @anubhavjana,

thanks for submitting this report. It looks like you're using Kubernetes in your setup to orchestrate Firecracker uVMs. We only provide support for Firecracker in this repository, and I'm personally not very familiar with Kubernetes. If you could reproduce the issue on an environment with Firecracker only that'd be ideal. Alternatively, please provide a detailed explanation of your complete setup.

In any case, I took a look and noticed you didn't set up any networking interface in Firecracker, nor a vsock. How is the communication between Kubernetes and the uVM supposed to work?

Furthermore, if you could provide some logs from your code, firecracker, and the uVM serial console, it would be helpful to understand what's happening.

Thanks,
Riccardo

[anubhavjana]

Hi @Manciukic , thanks for getting back. So my main motive is to have a firecracker pod up and running on kubernetes and then execute python script inside it. For this, I have already installed pip3 and python3 in the .ext4 filesystem that I am using.

This is the portion of the firecracker setup inside the yaml definition (you can consider that these are the steps that I would have also used on the bare metal host)

echo "--------Starting Firecracker VM--------"
              TAP_DEV="tap0"
              TAP_IP="172.16.0.1"
              MASK_SHORT="/30"

              # Setup network interface
              ip link del "$TAP_DEV" 2> /dev/null || true
              ip tuntap add dev "$TAP_DEV" mode tap
              ip addr add "${TAP_IP}${MASK_SHORT}" dev "$TAP_DEV"
              ip link set dev "$TAP_DEV" up

              # Enable IP forwarding
              echo 1 > /proc/sys/net/ipv4/ip_forward
              iptables -P FORWARD ACCEPT

              # Identify the host's primary network interface
              HOST_IFACE=$(ip -j route list default | jq -r '.[0].dev')
              echo "----------- Host Interface: $HOST_IFACE --------------"

              # Flush and reapply NAT rules
              iptables -t nat -D POSTROUTING -o "$HOST_IFACE" -j MASQUERADE 2>/dev/null || true
              iptables -t nat -A POSTROUTING -o "$HOST_IFACE" -j MASQUERADE
              iptables -A FORWARD -i "$TAP_DEV" -o "$HOST_IFACE" -j ACCEPT
              iptables -A FORWARD -i "$HOST_IFACE" -o "$TAP_DEV" -m state --state RELATED,ESTABLISHED -j ACCEPT

              SOCKET_PATH="/run/firecracker-${POD_NAME}.sock"
              rm -f $SOCKET_PATH  # Ensure no stale socket
              /usr/local/bin/firecracker --api-sock $SOCKET_PATH > /var/lib/firecracker.log 2>&1 &

              FC_PID=$!
              echo "-------- Firecracker started with PID: $FC_PID on socket: $SOCKET_PATH --------"

              # Configure Firecracker VM
              curl --unix-socket $SOCKET_PATH -X PUT "http://localhost/boot-source" \
                -H "Content-Type: application/json" \
                -d '{
                      "kernel_image_path": "/var/lib/firecracker-containerd/runtime/hello-vmlinux.bin",
                      "boot_args": "console=ttyS0 reboot=k panic=1 pci=off nomodules selinux=0 quiet loglevel=0"
                  }'

              echo "-------- Firecracker configured with kernel_image_path AND boot_args --------"

              # Attach root filesystem
              curl --unix-socket $SOCKET_PATH -X PUT "http://localhost/drives/rootfs" \
                -H "Content-Type: application/json" \
                -d '{
                      "drive_id": "rootfs",
                      "path_on_host": "/var/lib/firecracker-containerd/runtime/ubuntu-24.04.ext4",
                      "is_root_device": true,
                      "is_read_only": false
                  }'
              echo "-------- Attached root filesystem successfully --------"

              # Attach writable volume for output
              curl --unix-socket $SOCKET_PATH -X PUT "http://localhost/drives/output" \
                -H "Content-Type: application/json" \
                -d '{
                      "drive_id": "output",
                      "path_on_host": "/var/lib/firecracker-containerd/runtime/linpack-fc-output.txt",
                      "is_root_device": false,
                      "is_read_only": false
                  }'

              echo "-------- Attached writable volume for output successfully --------"
              FC_MAC="06:00:AC:10:00:02"

              # Set network interface
              curl --unix-socket $SOCKET_PATH \
              -X PUT 'http://localhost/network-interfaces/eth0' \
              -H 'Accept: application/json' \
              -H 'Content-Type: application/json' \
              -d '{
                  "iface_id": "eth0",
                  "guest_mac": "06:00:AC:10:00:02",
                  "host_dev_name": "tap0"
                }'

              

              sleep 0.1s
              # Start VM
              curl --unix-socket $SOCKET_PATH -X PUT "http://localhost/actions" \
                -H "Content-Type: application/json" \
                -d '{
                      "action_type": "InstanceStart"
                  }'

              echo "--------Firecracker VM started!--------"
              sleep 2s  # Increase wait time for VM boot

On deploying it, here are the logs:

Here is the output of ps aux from host.

 curl --unix-socket /run/firecracker-fc-test-5bb57cc476-cwqzm.sock -X GET "http://localhost/machine-config"

{"vcpu_count":1,"mem_size_mib":128,"smt":false,"track_dirty_pages":false}

Ran the following from the pod container where FC is supposed to boot up.

ip a show tap0
iptables -t nat -L -v

3: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 96:47:d9:d2:db:17 brd ff:ff:ff:ff:ff:ff
    inet 172.16.0.1/30 scope global tap0
       valid_lft forever preferred_lft forever
    inet6 fe80::9447:d9ff:fed2:db17/64 scope link 
       valid_lft forever preferred_lft forever
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all  --  any    eth0    anywhere             anywhere     

I have run the following command to setup tap device on the host

sudo ip tuntap add dev tap0 mode tap
sudo ip link set tap0 up
sudo ip addr add 192.168.100.1/24 dev tap0

Finally this is the firecracker boot log inside the pod container

So, could you please these logs and help me with how can I run a python script inside the microVM from outside the VM, say host. @Manciukic

[anubhavjana]

Hi @Manciukic @ShadowCurse - can i get some support on this ? This is a little bit on priority.