Skip to content
BranchesTags

Repository files navigation

Project Aether

Autonomous, decentralized, multi-tenant compute — a pure-Linux replacement for VMware.

License: Apache 2.0 Language: Rust Infrastructure: Pure Linux / KVM Status: Active Development

Aether turns an enterprise blade chassis — such as an HPE BladeSystem c7000 — into a dense, multi-tenant private cloud, with no proprietary management stack. It replaces VMware vCenter and vSphere DRS with a zero-dependency control plane written in Rust, built on a decentralized Reverse-Bidding Architecture that delivers "just enough orchestration" at sub-100ms boot times and near-zero hypervisor overhead.

Where VMware centralizes scheduling in a heavyweight, license-locked server, Aether pushes the decision to the edge: every blade bids for the workloads it can run best, and the winner boots the VM. No central master. No stale global database. No nested-virtualization tax.

Why Aether

Enterprise virtualization is expensive, opaque, and centralized. A single vCenter outage can freeze an entire fleet, DRS decisions lean on a database that is always slightly out of date, and every feature sits behind a license. Aether is built on the opposite bet:

  • Decentralized by design. Placement is a local decision made by each blade in a 250ms auction — there is no central scheduler to fall behind or fall over.
  • Pure Linux substrate. Aether runs QEMU-KVM directly on bare-metal Linux for both ephemeral and persistent VMs. Immutable Dockerfile-built guest images become qcow2 bases; Cloud-Init is only the per-instance metadata layer.
  • Zero-dependency Rust. A small set of statically-linked daemons (unsafe code is forbidden workspace-wide), not a sprawling Java/appliance estate.
  • GitOps as the source of truth. Desired state lives in Kubernetes CRDs synced by FluxCD; northbound workload requests also use a small durable journal so retries and process restarts converge safely.
  • Hardware-honest HA. Failover uses out-of-band power fencing (STONITH) through the chassis management controller, not a software guess about whether a node is really dead.

The VMware Replacement Matrix

Aether replaces heavy, stateful proprietary components with lightweight, decentralized open-source equivalents:

Legacy VMware Component Aether Open-Source Replacement
vCenter Server Kubernetes Operator synced via FluxCD, with a replay-safe workload journal
vSphere DRS Autonomous, decentralized gRPC Reverse-Bidding, calculated locally on each blade
vSphere HA Out-of-band hard-power fencing (STONITH) via the HPE iLO 5 Redfish API
vSphere Distributed Switch Physical partitioning via HPE Virtual Connect Flex-10 MLAG & native Linux bridges
VMware VMFS / vSAN Local ZFS on Linux (ZoL) Volumes (ZVOLs) & thin-provisioned LVM pools

Project Status

Aether is under active development, and the foundational layers already run under test:

  • ✅ Working today: the mTLS-secured gRPC substrate, the reverse-bidding auction engine, the deterministic tie-breaker, native QEMU-KVM lifecycle, and the versioned OCI-to-KVM image contract. The older Firecracker driver remains in-tree for compatibility experiments but is not selected by the primary workload path.
  • 🚧 In active build-out: storage slicing (ZFS/iSCSI/CSI), live migration (the migration socket is complete; block/memory transfer are being hardened), and network tagging.
  • 🗺️ On the roadmap: out-of-band fencing & HA, the developer CLI and guest operations, multi-vendor hardware abstraction, and Cluster API integration.

See Delivery Stages & Product Roadmap for the detailed picture.

How It Works

Cluster architecture

Aether splits a physical blade chassis (e.g., 16 slots, 640 CPU cores, 4 TB RAM) into two logical pools, each running a minimal bare-metal Linux install and the aetherd node daemon:

graph TD
    A[GitOps Repo / FluxCD] -->|Applies Custom CRDs| B[Kubernetes Utility Cluster]
    B -->|Aether Aggregator Operator| C[Control Bus - VLAN 10]

    C -->|gRPC Reverse-Bid Request| VS[Virtual Connect Switch]

    subgraph "HPE c7000 Chassis"
        direction TB

        subgraph "Pool Alpha: Compute (Slots 1-8)"
            direction LR
            C1[Blade 1: aetherd] -->|Spawns| VM1[KVM VM + qcow overlay]
            C2[Blade 8: aetherd] -->|Spawns| VM2[KVM VM + qcow overlay]
        end

        subgraph "Pool Beta: Infrastructure (Slots 9-16)"
            direction LR
            S1[Blade 9: aetherd] -->|Runs| KVM1[QEMU/KVM VM]
            S2[Blade 16: aetherd] -->|Runs| KVM2[QEMU/KVM VM]
        end

        VS --> C1
        VS --> C2
        VS --> S1
        VS --> S2
    end

    classDef compute fill:#1f2937,stroke:#10b981,stroke-width:2px,color:#fff;
    classDef storage fill:#1f2937,stroke:#3b82f6,stroke-width:2px,color:#fff;
    classDef switch fill:#1f2937,stroke:#eab308,stroke-width:2px,color:#fff;
    class C1,C2,VM1,VM2 compute;
    class S1,S2,KVM1,KVM2 storage;
    class VS switch;
Loading

Pool Alpha — Compute Blades (Slots 1–8) Ephemeral developer environments and multi-tenant workers. aetherd uses native QEMU-KVM with immutable, digest-cached base images and disposable qcow2 overlays. This is the same lifecycle authority and image format used by the infrastructure pool, with different placement and storage policy.

Pool Beta — Storage & Infrastructure Blades (Slots 9–16) Long-lived persistent VMs, production database replicas, and Kubernetes control/worker nodes. These blades run full QEMU-KVM and back block storage with ZFS on Linux (ZVOLs), enabling inline compression, thin provisioning, and near-instant atomic snapshot cloning.

Topology note: coordination is a star, not a mesh. Each aetherd holds a single mTLS gRPC channel to the Aggregator — there is no Raft, no gossip, and no quorum between blades. The only blade-to-blade traffic is live migration between a specific source and target.

The reverse-bidding loop

Instead of a central scheduler pushing workloads onto nodes from a stale global view, Aether runs a decentralized, pull-based marketplace:

sequenceDiagram
    participant GitOps as GitOps / FluxCD
    participant Aggregator as Aether Aggregator (Operator)
    participant Node1 as Blade Daemon (aetherd)
    participant Node2 as Blade Daemon (aetherd)

    GitOps->>Aggregator: Apply AetherVirtualDeployment Spec
    Aggregator->>Node1: Broadcast: RequestReverseBid(Spec) via gRPC
    Aggregator->>Node2: Broadcast: RequestReverseBid(Spec) via gRPC
    Note over Node1,Node2: 250ms Auction Window Opens
    Node1->>Node1: Evaluate local CPU & Memory channel bandwidth
    Node2->>Node2: Evaluate local CPU & Memory channel bandwidth
    Node1->>Aggregator: Return Bid Score (e.g., 850)
    Node2->>Aggregator: Return Bid Score (e.g., 910)
    Note over Node1,Node2: 250ms Auction Window Closes
    Aggregator->>Aggregator: Select highest bidder (Node 2)
    Aggregator->>Node2: Dispatch: ExecuteProvisioning(Spec)
    Node2->>Node2: Clone storage, compile Cloud-Init drive, & Boot hypervisor
Loading
  1. Workload intent broadcast. The Aggregator receives a declarative request via GitOps and broadcasts the spec (CPU quotas, memory bytes, storage boundaries, tenant mappings) to all registered blade daemons over secure gRPC.
  2. Autonomous telemetry evaluation. Each blade queries its local kernel parameters — CPU task congestion, memory channel bandwidth, and drive wear leveling.
  3. The reverse-bid response. Nodes compute a score from 1 to 1000, returning -1 if they cannot safely host the instance without degrading current SLAs. Healthy nodes respond inside a strict 250ms convergence window.
  4. Deterministic convergence. The Aggregator accepts the highest bid. On ties, a multi-tier tie-breaker (chassis thermal layout, adjacent-slot density, SSD write wear) picks a winner deterministically. The winning node clones its local volume, compiles a NoCloud Cloud-Init drive in memory, and boots the hypervisor.

Quickstart

Aether is a Cargo workspace and uses just as a task runner.

# Build the entire workspace
cargo build --workspace

# Run the full test suite
just test            # cargo test -- --nocapture
just nt              # cargo nextest run (faster, parallel)

# Lint / pre-commit checks (rustfmt, clippy, ruff, mypy)
just setup-hooks     # one-time: install pre-commit hooks
just qa              # run all checks across the repo

# Coverage report (tarpaulin + markdown + threshold)
just coverage

Native KVM image contract

runtime_class = "aether-kvm" selects Aether's primary node runtime. An image URI such as oci://ghcr.io/microscaler/aether-kvm-ubuntu:latest resolves to a four-file bundle (image.json, rootfs.qcow2, vmlinuz, and initrd.img). aetherd streams the OCI layers to disk, verifies their registry digests, caches the bundle by immutable manifest digest, and creates one writable qcow2 overlay per workload. QEMU boots the matching kernel and initramfs directly. Native bidding fails closed unless the aetherd process can open /dev/kvm read/write and execute both qemu-system-x86_64 and qemu-img.

The guest filesystem is maintained as an ordinary Dockerfile in microscaler/cylon-images; this keeps packages, services, and the compatible cylon-skills registry declarative and testable. The NoCloud cidata ISO only carries instance identity and later per-instance configuration.

See Native KVM image contract.

Optional Cylon compatibility

The Resurrection Hub uses the Aggregator's idempotent EnsureWorkload, GetWorkload, and DeleteWorkload RPCs. The Aggregator persists desired and observed state, runs the placement auction, and dispatches the winning request to aetherd. For runtime_class = "cylon-far", aetherd delegates physical lifecycle operations to the Cylon service on that same node. The Hub must not also call that node-local service directly; Aether is the lifecycle authority.

Each Aggregator and aetherd process requires shared trust and token material:

  • AETHER_CA_CERT_PATH, AETHER_SERVER_CERT_PATH, and AETHER_SERVER_KEY_PATH configure its inbound mTLS identity.
  • AETHER_CLIENT_CERT_PATH and AETHER_CLIENT_KEY_PATH configure its outbound mTLS identity.
  • AETHER_TOKEN_SECRET_PATH points to the same token secret on Aggregator and nodes. The file must contain at least 32 bytes.

Aggregator deployment settings are AETHER_LISTEN_ADDR, AETHER_NODE_TLS_SERVER_NAME, and AETHER_WORKLOAD_STATE_PATH (default /var/lib/aether/workloads.json). The JSON store provides atomic durability for a single active Aggregator; multi-Aggregator deployments need a shared store or leader election before they can provide the same admission guarantee.

Node settings are AETHER_NODE_ID, AETHER_POOL, AETHER_LISTEN_ADDR, AETHER_GRPC_ENDPOINT, AETHER_AGGREGATOR_ENDPOINT, and AETHER_AGGREGATOR_TLS_SERVER_NAME. Native image paths are configured with AETHER_IMAGE_CACHE_DIR, AETHER_WORKLOADS_DIR, and AETHER_QEMU_IMG_BIN. Private registries use AETHER_OCI_USERNAME with AETHER_OCI_PASSWORD_FILE; credentials never travel in workload intent. Registry transport is HTTPS by default. Loopback registries may use HTTP; other development registries must be named exactly in the comma-separated AETHER_OCI_INSECURE_REGISTRIES allowlist (including the port).

Enabling the optional Cylon adapter additionally requires AETHER_CYLON_FAR_ENDPOINT. HTTPS endpoints use AETHER_CYLON_FAR_CA_CERT_PATH, AETHER_CYLON_FAR_CLIENT_CERT_PATH, AETHER_CYLON_FAR_CLIENT_KEY_PATH, and AETHER_CYLON_FAR_TLS_SERVER_NAME. On the Cylon process set CYLON_LIFECYCLE_AUTHORITY=aether; this disables its legacy Resurrection Hub registration/watchdog so it cannot mutate VM state outside Aether. Cylon persists its subordinate runtime registry at CYLON_VM_STATE_PATH.

The adapter endpoint is node-local Cylon (https://127.0.0.1:50051 in the current host daemon). It is a compatibility path for Cylon-specific FAR mechanics, not the default Aether VM implementation. During bidding, a node that does not have the exact requested runtime adapter returns a negative bid and cannot win initial placement or recovery.

The optional cylon-far runtime JSON accepts only workspace_project, workspace_git_url, workspace_branch, llm_router_url, and egress_billing_account_id. URLs must not embed credentials. Git credentials remain node-local (for example, in a credential helper); raw tokens are rejected at both API boundaries.

See Cylon FAR compatibility for the cylon-images, cylon-skills, PKI, and host-provisioning contract.

Repository Layout

The workspace is composed of five crates:

Crate Role
aetherd The per-blade node daemon, in a compute/infra or storage role: native QEMU-KVM lifecycle, OCI KVM image materialization, storage (ZFS/iSCSI, plus iSCSI target export + ZVOL replication in the storage role), Linux-bridge networking, live migration, Cloud-Init, telemetry, heartbeat, and the local bidding algorithm.
aether-aggregator The Kubernetes operator and workload controller: durable northbound intent, pool-aware node registry, bid scheduler, deterministic tie-breaker, HA fencing + recovery (re-auction), stable-MAC identity, storage provisioning/discovery, CSI storage driver, and HPE Virtual Connect networking.
aether-auth Shared mTLS handshakes and single-use ephemeral attestation tokens.
aether-fence Out-of-band STONITH fencing via the iLO 5 Redfish API (iLO/iDRAC drivers built; wired into the HA loop).
pact-mock-server Contract-test mock server for validating gRPC API boundaries.

gRPC contracts live in proto/; design documents live in docs/.

Delivery Stages & Product Roadmap

The programme is organized into incremental stages that safely migrate a fleet off VMware:

[ Stage 1: API & Proto ] ──► [ Stage 2: Auction loop ] ──► [ Stage 3: Native KVM ]
                                                                     │
┌────────────────────────────────────────────────────────────────────┘
▼
[ Stage 4: ZFS & VC HAL ] ──► [ Stage 5: Live Migration ] ──► [ Stage 6: iLO Fencing / HA ]
                                                                     │
┌────────────────────────────────────────────────────────────────────┘
▼
[ Stage 7: Dev CLI & Vsock ] ─► [ Stage 8: Multi-Vendor HAL ] ─► [ Stage 9: Cluster API (CAPI) ]

🟩 Stage 1 — Core API & Rust Substrate (Active)

  • Compile-target gRPC schemas under aether.proto.
  • Cargo workspace layout (Aggregator, Daemon, Auth, Fencing).
  • Baseline Mutual TLS (mTLS) socket handshakes.

🟩 Stage 2 — Stateless Reverse-Bidding & Scheduling (Active)

  • In-memory NodeRegistry and WorkloadPlacement state tables in the Aggregator.
  • Asynchronous 250ms broadcast bidding convergence loop.
  • Local host telemetry checks in aetherd (CPU loadavg, memory channel pressure).

🟦 Stage 3 — Native KVM Engine (In Progress)

  • QEMU-KVM management and QMP lifecycle in aetherd.
  • Dockerfile-built OCI guest bundles with digest cache and per-VM qcow2 overlays.
  • Direct kernel/initramfs boot plus dynamic NoCloud cidata metadata in host memory.

🟦 Stage 4 — Storage Slicing & Net Tagging (In Progress)

  • Local ZFS on Linux (ZVOL) snapshot cloning with near-instant provisioning.
  • democratic-csi integration for Kubernetes persistent storage.
  • Virtual Connect Flex-10 hardware VLAN trunk tagging.

🟦 Stage 5 — Live Migration & Auto-Convergence (In Progress)

  • QEMU drive-mirror + NBD block replication for local disk migrations.
  • Iterative memory pre-copy over TCP migration sockets.
  • Auto-Converge vCPU throttling to guarantee convergence under active write loads.

🟦 Stage 6 — Out-of-Band Fencing & HA (In Progress)

  • Redfish STONITH client (aether-fence) targeting HPE iLO 5 / iDRAC endpoints. ✅ built + tested
  • Node heartbeat deadman loop (5s beat, 15s prune-timeout failover). ✅ wired
  • STONITH-and-recover workflow: prune → exclude storage nodes → out-of-band corroboration veto → power-off → re-auction the orphaned VMs. ✅ built (real iLO/OneView corroborator pending)
  • Disaggregated storage node: ZVOL provisioning, iSCSI target export, and on-demand volume provisioning from placement. ✅ built
  • ZFS asynchronous volume replication for DR (RPO 5m). ✅ engine built (zrepl/array integration deferred behind a transport seam)
  • Stable-MAC identity handed off to DCops NetBox/IPAM so a recovered VM keeps its IP. ✅ built

See the HA, storage-node, and network-identity deep-dives.

░░ Stage 7 — Developer CLI & Guest Operations (Planned)

  • The aether developer CLI client.
  • Zero-network guest access (aether shell / aether exec) via QEMU Guest Agent sockets.
  • Local directory passthrough via VirtioFS.

░░ Stage 8 — Multi-Vendor Hardware Abstraction (Planned)

  • Abstract ChassisManager and MidplaneNetworkManager interfaces for Dell PowerEdge MX7000 and IBM/Lenovo Flex chassis.

░░ Stage 9 — Cluster API (CAPI) Integration (Planned)

  • Kubernetes-native Cluster API provider (cluster-api-provider-aether) reconciling AetherMachine resources via AetherVirtualDeployment CRDs.
  • Expose blade slots as native FailureDomains inside AetherCluster for CAPI node distribution.
  • Dynamic IP discovery via DHCP snooping and QEMU Guest Agent queries.

Further Reading

Licensed under Apache 2.0.

About

No description, website, or topics provided.

Resources

Readme
License

Code of conduct

Code of conduct

Contributing

Contributing
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages