CubeSandbox on Kubernetes

[wwthw]

CubeSandbox supports K8s management, delivering a friendly experience for users within the K8s ecosystem.

With the resource management capability of Kubernetes, both the control plane and compute nodes can be more elastic. This delivers a friendly experience for users in the Kubernetes ecosystem.

K8s boasts a large user ecosystem. the capability of managing CubeSandbox sandboxes via K8s, which will greatly expand CubeSandbox's ecosystem.

[wwthw]

┌─────────────────────────────────────────────────────────────────────────────┐
│                         Kubernetes Cluster                                   │
│                                                                              │
│  ┌───────────────────────  Access Layer  ────────────────────────────────┐  │
│  │                                                                        │  │
│  │  ┌─────────────────┐         ┌──────────────────┐                     │  │
│  │  │    CubeProxy     │         │     CubeAPI       │                     │  │
│  │  │  (DaemonSet/     │         │  (Deployment/     │                     │  │
│  │  │   hostNetwork)   │         │   ClusterIP)      │                     │  │
│  │  │                   │         │                    │                     │  │
│  │  │  :443 TLS ◄────── Data    │  :3000 ◄─────── Control               │  │
│  │  │  :80   HTTP       Plane   │  (API Server)      Plane               │  │
│  │  │                   │         │                    │                     │  │
│  │  │  Host-based:      │         │  POST /sandboxes  │                     │  │
│  │  │  <port>-<id>      │         │  GET  /sandboxes  │                     │  │
│  │  │  .cube.app        │         │  DELETE /sandboxes│                     │  │
│  │  │                   │         │  ...               │                     │  │
│  │  │  Path-based:      │         │                    │                     │  │
│  │  │  /sandbox/<id>/   │         │                    │                     │  │
│  │  │  <port>/          │         │                    │                     │  │
│  │  └─────────┬─────────┘         └─────────┬────────┘                     │  │
│  │            │                              │                              │  │
│  │            │ Redis                        │ HTTP POST                    │  │
│  │            │ (bypass_host_proxy:<id>)     │ /cube/sandbox                │  │
│  └────────────┼──────────────────────────────┼─────────────────────────────┘  │
│               │                              │                               │
│  ┌────────────┼───────  Control Plane  ──────┼─────────────────────────────┐  │
│  │            │                              │                              │  │
│  │  ┌─────────▼─────────┐         ┌─────────▼─────────┐                   │  │
│  │  │    Redis           │         │   CubeMaster       │                   │  │
│  │  │  (StatefulSet/     │         │  (Deployment/       │                   │  │
│  │  │   ClusterIP)       │         │   ClusterIP)        │                   │  │
│  │  │                    │         │                      │                   │  │
│  │  │  沙箱元数据存储     │         │  :8089 HTTP          │                   │  │
│  │  │  bypass_host_proxy │         │                      │                   │  │
│  │  │  :<sandboxID>      │         │  Scheduler → 选节点  │                   │  │
│  │  │                    │         │  gRPC → PodIP:9999   │                   │  │
│  │  └────────────────────┘         └──────────┬─────────┘                   │  │
│  └───────────────────────────────────────────┼─────────────────────────────┘  │
│                                              │ gRPC: PodIP:9999              │
│                                              │                                │
│  ┌───────────────────────────────────────────┼──── Data Plane ──────────────┐  │
│  │                                           │                              │  │
│  │  ┌─────────────────── cubelet-control Pod ┼─────────────────────────┐    │  │
│  │  │                    (DaemonSet, per Node) │                        │    │  │
│  │  │                                          │                        │    │  │
│  │  │  ┌──────────────────┐  ┌─────────────────▼───────────────────┐   │    │  │
│  │  │  │  network-agent   │  │           cubelet                    │   │    │  │
│  │  │  │  (容器网络/CNI)  │  │         (容器网络/CNI)               │   │    │  │
│  │  │  │                  │  │                                      │   │    │  │
│  │  │  │  :19090 health   │  │  :9999 gRPC (PodIP直连)             │   │    │  │
│  │  │  │  hostPort:19090  │  │                                      │   │    │  │
│  │  │  │                  │  │  ┌─── RunCubeSandbox ───────────┐   │   │    │  │
│  │  │  │  Unix Socket:    │  │  │  ┌─────────────────────┐    │   │   │    │  │
│  │  │  │  /tmp/cube/      │  │  │  │    Sandbox (VM)      │    │   │   │    │  │
│  │  │  │  network-agent-  │  │  │  │    KVM + CubeShim    │    │   │   │    │  │
│  │  │  │  grpc.sock       │  │  │  │    NEED_SET_NET      │    │   │   │    │  │
│  │  │  │                  │  │  │  │    → host netns       │    │   │   │    │  │
│  │  │  │  eBPF/Tap/TCP    │  │  │  │  └─────────────────┘    │   │   │    │  │
│  │  │  │  (host netns     │  │  │  │    运行在本Pod内       │    │   │   │    │  │
│  │  │  │   via netlink    │  │  │  └──────────────────────────┘   │   │    │  │
│  │  │  │   HandleAt)      │  │  │                                  │   │    │  │
│  │  │  └──────────────────┘  │  │  ←→ network-agent via Unix     │   │    │  │
│  │  │                         │  │      Socket /tmp/cube/          │   │    │  │
│  │  │  共享: /tmp/cube (hostPath)                                  │   │    │  │
│  │  │  特权: privileged: true                                       │   │    │  │
│  │  │  挂载: /dev/kvm, /dev/net/tun等                               │   │    │  │
│  │  └──────────────────────────────────────────────────────────────┘   │    │  │
│  │                                                                     │    │  │
│  │  ┌── Worker Node ──┐  ┌── Worker Node ──┐  ┌── Worker Node ──┐     │    │  │
│  │  │ cubelet-control │  │ cubelet-control │  │ cubelet-control │     │    │  │
│  │  │ Pod (全量副本)   │  │ Pod (全量副本)   │  │ Pod (全量副本)   │     │    │  │
│  │  └─────────────────┘  └─────────────────┘  └─────────────────┘     │    │  │
│  └─────────────────────────────────────────────────────────────────────┘  │  │
│                                                                            │  │
│  ┌───── Support Services ─────────────────────────────────────────────┐    │  │
│  │  WebUI (Deployment) │ Redis (StatefulSet)   │ mysql (StatefulSet)  │    │  │
│  └─────────────────────────────────────────────────────────────────────┘  │  │
└─────────────────────────────────────────────────────────────────────────────┘

Worker Node Deployment architecture

┌─────────────────────── Worker Node ──────────────────────────────┐
│                                                                  │
│  ┌────────────── cubelet-control DaemonSet Pod ───────────────┐  │
│  │  (Pod, nodeSelector: cube-role=control)                                                     │  │
│  │                                                            │  │
│  │  ┌───────────── network-agent ─────────────┐               │  │
│  │  │  Container Network (CNI Pod IP)         │               │  │
│  │  │                                         │               │  │
│  │  │  Process:                               │               │  │
│  │  │    network-agent (主进程)                │               │  │
│  │  │    + eBPF/Tap/TCP 数据面                 │               │  │
│  │  │      (netlink.HandleAt → host netns)    │               │  │
│  │  │                                         │               │  │
│  │  │  Listening:                             │               │  │
│  │  │    Unix: /tmp/cube/network-agent-       │               │  │
│  │  │         grpc.sock  (gRPC)               │               │  │
│  │  │    Unix: /tmp/cube/network-agent-       │               │  │
│  │  │         tap.sock   (Tap FD)             │               │  │
│  │  │    TCP:  127.0.0.1:19090 (health)       │               │  │
│  │  │                                         │               │  │
│  │  │  hostPort: 19090 → NodeIP可达            │               │  │
│  │  │                                         │               │  │
│  │  │  VolumeMounts:                          │               │  │
│  │  │    /tmp/cube          (hostPath共享)     │               │  │
│  │  │    /dev/net/tun       (hostPath)        │               │  │
│  │  │    config/config.toml (ConfigMap)        │               │  │
│  │  │    state/             (hostPath)         │               │  │
│  │  │    /data/log/         (hostPath)         │               │  │
│  │  │                                          │               │  │
│  │  │  Init Sequence (start-network-agent.sh): │               │  │
│  │  │    1. 等待hostPath卷就绪                  │               │  │
│  │  │    2. 初始化eBPF程序(需/sys/fs/bpf)        │               │  │
│  │  │    3. 创建cube网络设备                     │               │  │
│  │  │    4. 准备Tap池(500个)                    │               │  │
│  │  │    5. 启动health probe                    │               │  │
│  │  │    6. 启动gRPC/Tap Socket监听              │               │  │
│  │  └───────────────────────────────────────────┘               │  │
│  │               ↕ Unix Socket /tmp/cube/                       │  │
│  │               ↕ shareProcessNamespace: true                  │  │
│  │                                                              │  │
│  │  ┌───────────── cubelet ───────────────────┐               │  │
│  │  │  Container Network (CNI Pod IP)         │               │  │
│  │  │                                          │               │  │
│  │  │  Process:                                │               │  │
│  │  │    cubelet (主进程)                       │               │  │
│  │  │                                          │               │  │
│  │  │  Listening:                              │               │  │
│  │  │    TCP: 0.0.0.0:9999 (gRPC)              │               │  │
│  │  │      ← CubeMaster直连PodIP:9999          │               │  │
│  │  │                                          │               │  │
│  │  │  Sandbox Runtime:                        │               │  │
│  │  │    ┌────────────────────────────┐        │               │  │
│  │  │    │  Sandbox 1 (VM process)    │        │               │  │
│  │  │    │  KVM → CubeShim(NEED_SET_  │        │               │  │
│  │  │    │  NET → host netns)         │        │               │  │
│  │  │    └────────────────────────────┘        │               │  │
│  │  │    ┌────────────────────────────┐        │               │  │
│  │  │    │  Sandbox 2 (VM process)    │        │               │  │
│  │  │    │  KVM → CubeShim            │        │               │  │
│  │  │    └────────────────────────────┘        │               │  │
│  │  │    ... (多沙箱 )                    │               │  │
│  │  │                                          │               │  │
│  │  │  VolumeMounts:                           │               │  │
│  │  │    /tmp/cube            (hostPath共享)   │               │  │
│  │  │    /dev/kvm             (hostPath)       │               │  │
│  │  │    /dev/net/tun         (hostPath)       │               │  │
│  │  │    /dev/vsock           (hostPath)       │               │  │
│  │  │    /sys/fs/bpf          (hostPath)       │               │  │
│  │  │    /sys/fs/cgroup       (hostPath)       │               │  │
│  │  │    config/config.toml   (ConfigMap)      │               │  │
│  │  │    dynamicconf/         (ConfigMap)      │               │  │
│  │  │    /data/cubelet        (hostPath)       │               │  │
│  │  │    /data/log/           (hostPath)       │               │  │
│  │  │    /data/cube-shim/     (hostPath)       │               │  │
│  │  │    /run/cube-containers (hostPath)       │               │  │
│  │  │                                          │               │  │
│  │  │  Init Sequence (start-cubelet.sh):       │               │  │
│  │  │    1. 轮询network-agent health           │               │  │
│  │  │       (127.0.0.1:19090, 阻塞等待就绪)     │               │  │
│  │  │    2. 注册节点到CubeMaster                │               │  │
│  │  │       POST /internal/meta/nodes/register │               │  │
│  │  │       上报PodIP作为HostIP                 │               │  │
│  │  │    3. 启动cubelet gRPC :9999              │               │  │
│  │  └────────────────────────────────────────────┘               │  │
│  │                                                              │  │
│  │  Pod Spec:                                                   │  │
│  │    privileged: true (both containers)                         │  │
│  └──────────────────────────────────────────────────────────────┘  │
│                                                                  │
│  Host Paths:                                                     │
│    /tmp/cube/          ← 容器间Socket通信                        │
│    /data/cubelet/      ← cubelet持久化数据                       │
│    /data/log/          ← 日志持久化                              │
│    /data/cube-shim/    ← VM磁盘存储                              │
│    /run/cube-containers ← 运行时状态                              │
│    /dev/kvm, /dev/net/tun,                                       │
│                                                                  │
└──────────────────────────────────────────────────────────────────┘

[wwthw]

• Should include CubeEgress(introduced in v0.4.0) in worker node.
• Should also mount /etc/cube/ca

I submitted a pull request based on my personal forked repository. Currently adapted the code up to around June 8th.
Please help review if it is feasible to upstream these Kubernetes-native features to the community.

CubeSandbox on Kubernetes, compute node image build and deploy.
wwthw#1

[kinwin-ustc]

In this deployment mode, if we update the Cubelet (the control surface component of the sandbox), does that mean all sandboxes need to be destroyed and rebuilt?

[wwthw]

In this deployment mode, if we update the Cubelet (the control surface component of the sandbox), does that mean all sandboxes need to be destroyed and rebuilt?

If we adopt isolated PID namespaces and container networking separated from the host, this problem will exist.

This problem should be designed based on actual business requirements. To guarantee uninterrupted operation of sandboxes throughout the upgrade process, you may adopt one of the deployment strategies provided below.

Solution A: Shared Host PID Namespace (hostPID: true)
Configure the cubelet Pod with hostPID: true so that the container shares the host's PID namespace. Since containerd-shim-cube-rs daemonizes via setsid() and becomes a child of PID 1 (init/systemd) on the host, it is outside the container's cgroup process tree. When the Pod is deleted, the kubelet only kills processes within the container's cgroup — the shim daemon survives because it has already detached into the host PID namespace.

spec:
hostPID: true
hostNetwork: true
privileged: true

Solution B: Separate containerd and Shim onto the Host

Decouple the data plane (containerd + shim + VM) from the control plane (cubelet). Run containerd and the shim as host-level systemd services, while cubelet runs as a lightweight Pod that connects to the external containerd via a Unix socket. When the cubelet Pod is upgraded or recreated, the data plane processes on the host remain untouched — sandboxes continue running independently.

Current:
cubelet Pod (container)
├── cubelet process (gRPC server + workflow engine)
├── containerd server (embedded, in-process)
├── containerd-shim-cube-rs (spawned by embedded containerd, daemonizes via setsid)
│ └── VMM thread (in-process)
│ └── MicroVM (KVM)
└── cube-agent (in-VM, vsock)

Proposed:
Host (systemd services):
├── containerd daemon (standalone systemd service)
│ └── containerd-shim-cube-rs (spawned by containerd, daemonizes via setsid)
│ └── VMM thread (in-process)
│ └── MicroVM (KVM)
└── cube-agent (in-VM, vsock)

Cubelet Pod (Kubernetes):
├── cubelet process (gRPC server + workflow engine)
└── connects to host containerd via /run/containerd/containerd.sock (hostPath volume)

[wwthw]

• Should include CubeEgress(introduced in v0.4.0) in worker node.
• Should also mount /etc/cube/ca

I submitted a pull request based on my personal forked repository. Currently adapted the code up to around June 8th. Please help review if it is feasible to upstream these Kubernetes-native features to the community.

CubeSandbox on Kubernetes, compute node image build and deploy. wwthw#1

updtae Kubernetes Deployment Configurations for CubeSandbox Control Plane

[kinwin-ustc]

Deploying Cube on K8S is in our plan, we have already been working on deploying Cube using Terraform (PR #629), welcome to contribute, let's expand the applicability of Cube together

[jamesxia20181001]

The core challenges for the managed operation of CubeSandbox in Kubernetes lie in the deployment and operation mode of CubeSandbox compute node components as well as their full lifecycle management strategies.

The solution previously proposed by wwthw adopts the Underlay technical route, which suffers from tight coupling with Kubernetes. This flaw tends to trigger a series of issues including resource scheduling conflicts between containers and sandboxes, and network stack conflicts across containers and sandboxes. We propose an optimized Overlay technical route with higher decoupling. A brief architectural diagram of the solution is provided below, and discussions from all participants are welcome to iterate and refine more practical implementation approaches.

+--------------------------------------------------------------------------------------------------+
|                                         CubeControlManger                                        |
|                                                                                                  |
|   +---------------------------+  +------------------------------+  +---------------------------+  |
|   |   CubeNodeController      |  |   CubeNodePoolController     |  |           ...             |  |
|   +---------------------------+  +------------------------------+  +---------------------------+  |
+--------------------------------------------------------------------------------------------------+


+--------------------------------------------------------------------------------------------------+
|                                          K8S Work Node                                           |
|                                                                                                  |
|  +----------------------+  +-----------------------------------------------+  +----------------+ |
|  |     Normal Pods      |  |               CubeNode Pods                   |  | CubeNodeManager| |
|  |                      |  |                                               |  |      Pod       | |
|  |                      |  |  +-----------------------------------------+  |  |                | |
|  |                      |  |  |              Containers                 |  |  |                | |
|  |                      |  |  |                                         |  |  |                | |
|  |                      |  |  |  +-------------+  +-------------+       |  |  |                | |
|  |                      |  |  |  |    Pause    |  |    Init     |       |  |  |                | |
|  |                      |  |  |  +-------------+  +-------------+       |  |  |                | |
|  |                      |  |  |                                         |  |  |                | |
|  |                      |  |  |  +-------------+  +-------------------+ |  |  |                | |
|  |                      |  |  |  | Network     |  | Active-standby    | |  |  |                | |
|  |                      |  |  |  | Agent       |  | CubeEgress*2      | |  |  |                | |
|  |                      |  |  |  +-------------+  +-------------------+ |  |  |                | |
|  |                      |  |  |                                         |  |  |                | |
|  |                      |  |  |  +-------------+  +-------------------+ |  |  |                | |
|  |                      |  |  |  | Cubelet     |  | Active-standby    | |  |  |                | |
|  |                      |  |  |  |             |  | cube-shims*2      | |  |  |                | |
|  |                      |  |  |  +-------------+  +-------------------+ |  |  |                | |
|  |                      |  |  +-----------------------------------------+  |  |                | |
|  |                      |  |                                               |  |                | |
|  |                      |  |  +-----------------------------------------+  |  |                | |
|  |                      |  |  | Shared pod-level network / UTS / IPC /  |  |  |                | |
|  |                      |  |  | PID namespaces                          |  |  |                | |
|  |                      |  |  |                                         |  |  |                | |
|  |                      |  |  |     +-------------------------------+   |  |  |                | |
|  |                      |  |  |     | Taps                          |   |  |  |                | |
|  |                      |  |  |     +-------------------------------+   |  |  |                | |
|  |                      |  |  |     +-------------------------------+   |  |  |                | |
|  |                      |  |  |     | Active-standby CubeVS*2       |   |  |  |                | |
|  |                      |  |  |     +-------------------------------+   |  |  |                | |
|  |                      |  |  |     +-------------------------------+   |  |  |                | |
|  |                      |  |  |     | eth0, eth1, ethN              |   |  |  |                | |
|  |                      |  |  |     +-------------------------------+   |  |  |                | |
|  |                      |  |  +-----------------------------------------+  |  |                | |
|  |                      |  |                                               |  |                | |
|  |                      |  |  +-----------------------------------------+  |  |                | |
|  |                      |  |  | Pod-level CPU and memory requests &    |  |  |                | |
|  |                      |  |  | limits                                |  |  |                | |
|  |                      |  |  +-----------------------------------------+  |  |                | |
|  |                      |  |                                               |  |                | |
|  |                      |  |  +-----------------------------------------+  |  |                | |
|  |                      |  |  | Shared pod-level emptydirs              |  |  |                | |
|  |                      |  |  +-----------------------------------------+  |  |                | |
|  |                      |  |                                               |  |                | |
|  |                      |  |  +-----------------------------------------+  |  |                | |
|  |                      |  |  | Shared pod-level SSD hostpaths or      |  |  |                | |
|  |                      |  |  | local SSD PVs                          |  |  |                | |
|  |                      |  |  +-----------------------------------------+  |  |                | |
|  |                      |  |                                               |  |                | |
|  +----------------------+  +-----------------------------------------------+  +----------------+ |
+--------------------------------------------------------------------------------------------------+

I. Solution Design Objectives

1. Achieve a loosely coupled architecture between CubeSandbox and Kubernetes
   1.1 Decouple and isolate three layers of networks: sandbox network, container network and host network from one another;
   1.2 Enable sandbox network forwarding via multiple container network plane interfaces of CubeNode, simplifying communication links between sandboxes and multi-plane container network services;
   1.3 Support flexible mixed deployment of workloads: normal Pods can be deployed on the same node alongside CubeNode Pods, and multiple CubeNode Pods can also be scheduled onto a single node.

2. Realize managed full lifecycle operations for CubeNode via Kubernetes extended CRDs
   2.1 Map each CubeNode to one Kubernetes Pod in a 1:1 manner for initial deployment and service launch;
   2.2 Support CubeNode Pod reconstruction and failure recovery. Combined with sandbox eviction policies, this accommodates CubeNode Pod reconstruction during host OS rolling upgrades and scenarios where in-place upgrades are unavailable;
   2.3 Support in-place cold upgrades for control-plane containers within CubeNode Pods;
   2.4 Support in-place hot upgrades for data-plane containers within CubeNode Pods;
   2.5 Support rolling gray-scale release and upgrade for CubeNode Pods;
   2.6 Enable automatic CubeNode provisioning by extending the CubeNodePool CRD. The controller automatically filters eligible nodes based on constraints such as available node resources and node labels to dynamically create CubeNode instances. Large-sized CubeNode Pods are prioritized to mitigate resource fragmentation caused by fixed resource specifications, and a single CubeNodePool can accommodate CubeNodes with varying resource configurations.

II. Core Solution Highlights

1. Define two new CRD resources: CubeNode and CubeNodePool, with dedicated controllers developed to implement resource orchestration;
2. With reference to the in-place cold upgrade capability of SidecarSet in OpenKruise, implement in-place cold updates for control-plane containers within CubeNode Pods.
3. With reference to the in-place hot upgrade control mechanism of OpenKruise SidecarSet, we implement non-stop hot upgrades for active-standby data-plane containers within CubeNode Pods.
4. With reference to OpenKruise Advanced StatefulSet, stable and ordered Pod identities are allocated to CubeNodes. Relying on the IPAM capability of container network plugins, fixed Pod IP addresses can be maintained throughout the entire lifecycle of CubeNodes.
5. Enforce overall CPU and memory upper limits for CubeNode Pods via Pod-level resource quotas, while CubeSandbox performs secondary resource scheduling at the sandbox granularity inside each CubeNode Pod;
6. Deploy CubeNodeManager Pods as DaemonSet workloads on each node to handle node-level operations including configuration validation, dynamic configuration hot-reloading, and cleanup of hostPath storage directories on the host;
7. Grant privileged permissions to CubeNode Pod containers on demand. Such permissions remain fully transparent to upper-layer user sandbox workloads, enabling controllable security risks.

III. Proposed Implementation & Iteration Timeline

Given the relatively high overall complexity of this solution, sufficient community review and validation are required, so phased rollout is recommended with priority given to iterating a minimum viable product (MVP):

Phase 1

Deploy CubeNode Pods via single-replica StatefulSets to retain stable Pod identities, facilitating fixed IP assignment via IPAM.
To simplify sandbox network implementation, only one CubeNode Pod is allowed per node in the initial phase.
All container image updates within CubeNode Pods are managed as node-level cold upgrades, with version iterations completed through sandbox eviction combined with rolling releases.

Phase 2

Decouple control-plane containers from data-plane containers in CubeNode Pods and support in-place cold upgrades for the control plane.
Implement the CubeNode and CubeNodePool controllers.

Phase 3

Enable active-standby containers for the data plane within CubeNode Pods and support in-place hot upgrades for the data plane.