[RFC] Per-sandbox eBPF observability for CubeSandbox

[zyl1121]

Motivation

CubeSandbox provides strong workload isolation through lightweight virtual machines. Each sandbox runs its workload inside a guest VM, but this VM boundary also limits visibility from the host. Host-side components cannot observe workload behavior such as process execution, file access, network activity, or privilege-sensitive operations without requiring the workload itself to add instrumentation.

A per-sandbox observability capability could provide this workload-level visibility while preserving isolation. It could support debugging, diagnostics, auditing, and exporting sandbox-scoped telemetry to external observability or security analysis systems.

For example, sandbox-scoped kernel records can complement application-level traces for workloads such as AI agents and code-execution services. Application traces describe intent and tool interactions, while kernel observation shows the actual process, file, network, and privilege-related activity triggered by those actions. Together they provide complementary views of what was requested and what actually happened inside the sandbox.

This discussion outlines a direction for developing per-sandbox eBPF observability in CubeSandbox. The current phase focuses on a proof of concept that validates the guest/host architecture and key technical choices. The longer-term goal is to provide a general observability capability for sandbox workloads. The same foundation could also support future policy enforcement or other workload security capabilities.

Current PoC Direction

The initial implementation uses eBPF programs in the guest kernel as the foundation for per-sandbox observability without requiring changes to the workload.

This direction is inspired by Ant Group's use of Kata Containers and eBPF:

• Ant Group's Cloud Workload Protection Platform Built with Kata Containers and eBPF

The Ant Group CWPP architecture uses a guest-side agent and eBPF programs to observe workload behavior and enforce policies. The current CubeSandbox work focuses on observation and telemetry collection. Once the observation architecture is validated, policy enforcement and other workload security capabilities can be explored on top of the same foundation.

The Ant Group CWPP architecture is a reference rather than a design to copy directly. CubeSandbox is not a Kubernetes-managed Kata environment. Its API-driven lifecycle includes fast creation from templates, pause/resume, snapshot/restore, rollback, and clone. The design must therefore preserve sandbox identity and observation state across these operations.

Initial Proof of Concept

The initial PoC will validate a complete but bounded observation path:

1. enable observation for a selected sandbox;
2. load a small set of eBPF programs in its guest kernel;
3. observe workload container activity while excluding guest management processes;
4. configure the selected observation categories from the host;
5. deliver versioned event records to a host-side consumer;
6. recover correctly from guest component restart and CubeSandbox lifecycle operations;
7. measure collection overhead, dropped records, buffering behavior, and transport cost.

The initial event set focuses on process lifecycle, file operations, and selected network and privilege-sensitive events. Network-related observation will be evaluated together with the existing network-agent responsibilities to avoid overlapping ownership.

The event set should remain small enough to measure, validate, and revise. The PoC should validate scope isolation, lifecycle correctness, and cost, and identify the work required to evolve the implementation. Public interfaces and production performance targets can be defined after these results are available.

Current Architecture Direction

The current direction separates observation from the existing workload lifecycle components:

• cube-agent remains the guest lifecycle supervisor and starts a dedicated guest observability component;
• the guest observability component owns BPF loading, attachment, event collection, and forwarding;
• host-side coordination and export remain outside Cubelet, keeping observability logic independent from sandbox lifecycle management;
• the guest runtime image provides the observability binary and BPF objects, while observation is activated and configured per sandbox.

Host/Guest Communication

Collection happens inside the guest, while configuration and event consumption happen on the host. The design therefore needs a host/guest control path and an event-delivery path.

CubeSandbox already has a vsock communication path. The current PoC will reuse it as the baseline so that the first milestone can focus on the observation model and lifecycle behavior rather than new transport infrastructure.

While developing this observability path, I also explored shared memory as a possible option for higher-volume event delivery. This resulted in #858, which introduces experimental, template-level opt-in ivshmem support with a minimal shared-memory ring example. Its same-host microbenchmarks showed higher throughput and lower latency than a raw vsock stream.

ivshmem remains a possible transport optimization after the basic observation path is working. Using it would require additional protocol, synchronization, notification, flow-control, and recovery mechanisms, which are outside the first PoC milestone. Keeping the event model independent of the transport will allow a later ivshmem backend to be evaluated without changing the overall observation architecture.

Areas for Feedback

The following directions are planned for the PoC. Feedback is particularly useful on risks, missing lifecycle cases, and constraints that would be costly to address after the architecture is implemented.

Observation scope

The initial PoC focuses on process lifecycle, file operations, and selected network and privilege-sensitive events. Network coverage will be defined alongside the existing network-agent responsibilities to avoid unclear or duplicated ownership.

Observation is scoped to workload containers rather than guest management processes. The initial implementation targets the current single-container model, while its identity model and protocol should leave room for future multi-container sandboxes.

Feedback requested:

• Are there other event types that would add high value without significantly expanding the first milestone?
• Are there scope-isolation or attribution risks missing from this model?

Architecture and lifecycle

The current direction keeps cube-agent as the lifecycle supervisor and uses a separate guest observability component for BPF and event handling. Host-side observation coordination and export are also kept outside Cubelet.

For the PoC, the guest component and BPF objects are included in the guest runtime image, with observation activated per sandbox. The architecture should leave room for independently versioned guest components and runtime updates later.

The design must handle startup failure, component restart, pause/resume, snapshot/restore, rollback, clone, and deletion without silently reusing stale identity or observation state.

Feedback requested:

• Do these component boundaries introduce operational or compatibility costs that should be addressed in the PoC?
• Are there packaging constraints, lifecycle cases, or recovery invariants missing from this direction?

Records and transport

The PoC will use a versioned event model and bounded in-memory buffering. If a consumer does not keep up, older unconsumed records may be discarded rather than allowing the buffer to grow without limit. Dropped records must be observable. A host-side export interface will allow events to be consumed outside the collection path.

Vsock is the baseline transport. Measurements of realistic event rates, payload sizes, latency, dropped records, and CPU cost will characterize its behavior and inform later transport optimization.

Feedback requested:

• Are there existing CubeSandbox conventions or external formats that should influence the event and export interfaces?
• Which workloads and event rates would provide representative transport measurements?

PoC Outcome

The expected outcome of the PoC is to validate the architecture and identify the remaining work required to build per-sandbox eBPF observability in CubeSandbox, including guest/host responsibilities, lifecycle handling, observation and export interfaces, and communication design. The capability can then be expanded and integrated incrementally based on the PoC results.