-
Notifications
You must be signed in to change notification settings - Fork 0
Implementing an Enterprise Proxmox
This document serves as a technical overview of the work completed for the Champlain colleges virtualization environment. This will not contain information in which would be designated to violate any NDA's present. It will discuss basic infrastructure, the migration from VMware vSphere to Proxmox VE, how the environment is used, and the operational considerations relevant to anyone working within it.
Champlain colleges virtual environment is a shared virtualization platform used to support ITS coursework at Champlain College. Students interact with deployed VMS in order to manage networked virtual machines for class assignments. Each course is allocated a set number of VMs [Between 3-12] appropriate to its curriculum, and students are provisioned with access to their respective resources at the start of each course section. The environment is designed to support multiple classes running concurrently, each with isolated network segments and independent VM pools.
The maximum number of VMs per student in a given class is 12. At a class size of 20 students that represents up to 240 VMs per section, and with multiple concurrent sections the environment has reached 1679 VMs at peak load. This scale drives the majority of infrastructure and storage decisions described below.
The environment runs on Proxmox VE, an open-source Type 1 hypervisor built on a modified Debian Linux kernel using KVM virtualization. A Type 1 hypervisor runs directly on bare metal hardware, partitioning physical CPU, RAM, and storage into isolated virtual machines without an intermediary operating system layer. A student team associated with the college installed the proxmox environment and dealt with student deployments.
The environment previously ran on VMware vSphere (ESXi), also a Type 1 hypervisor, which managed virtualization through ESXi on each host and vCenter Server as a centralized management plane. vSphere is a commercial product with strong enterprise tooling, official support, and a streamlined GUI. The shift to Proxmox was driven by the removal of education licensing for vSphere, which made continued use cost-prohibitive.
Proxmox offers several meaningful advantages in this context: it is free in its community edition, supports up to 32 physical nodes and 50,000 VMs per cluster, provides a broad range of storage backends, and supports both full KVM virtual machines and lightweight LXC Linux containers. 21 Proxmox hosts are grouped into a cluster and presented as a unified datacenter in the web interface, without requiring a separate management appliance.
The primary tradeoffs relative to vSphere are a steeper learning curve for advanced configuration (much of which must be done via CLI rather than GUI), gaps in official documentation, and the need for custom tooling to support mass deployments at the scale required by a multi-class academic environment.
The active directory domain created allows for identity and access control for the environment. Unlike vSphere, which automatically syncs with Active Directory, Proxmox requires manual re-synchronization after any AD user or group changes. Group permissions in Proxmox are mapped to AD groups to enforce least-privilege access, ensuring students can only interact with the VMs and resources assigned to them for their specific course. Five domain controller instances are distributed across the cluster for redundancy, with FSMO roles held by the primary DC.
Proxmox inherits the Linux kernel's OOM (Out of Memory) Killer, a mechanism that terminates running processes — including VMs — when available RAM is exhausted. This is a meaningful operational consideration during peak load when many students are simultaneously running resource-intensive VMs. RAM allocation per node and per class must be carefully planned to avoid unexpected VM terminations mid-session. This has been shown to also cause I/O Errors on the interface itself.
The cluster runs on rack mounted servers with Proxmox installed directly on each via the iDRAC interface allowing installation and administration without requiring a locally connected display or keyboard. Servers are connected redundantly to two separate power sources and bonded to switches via LACP (Link Aggregation Control Protocol) for combined throughput and redundancy. Two Ubiquiti switches and one UDM Pro handle the physical switching layer.
The cluster was originally planned for 21 nodes. PVE1, PVE8, and PVE21 are non-operational following issues encountered during initial setup, leaving 18 active nodes. Each server having died during our stress tests the summer prior to classes beginning. Information related to server stats will not be specified.
Storage in the environment uses thin provisioning within a shared NFS share. This is essential at this scale: when hundreds of VMs are deployed from the same base templates, thin provisioning prevents the immediate consumption of terabytes of storage for space that may never actually be used.
Node storage is local by default in Proxmox, meaning each server's disks are managed independently. To supplement this and address the overall storage capacity demands of peak class deployment, NFS (Network File System) shared storage is being implemented, connected directly via the server switch. Shared storage also enables live migration of VMs between nodes without needing to physically copy disk data, which is important for load balancing across the cluster.
The environment has also implemented iSCSI (Internet Small Computer System Interface), a protocol that allows block-level storage commands to be sent over a network to storage devices on secondary servers. Proxmox does not include iSCSI support by default — it must be installed manually. In some configurations, iSCSI also requires multipathing to be enabled, which routes storage traffic through multiple network paths simultaneously for redundancy and throughput.
VM disk images are stored in one of two primary formats. .qcow2 is the native QEMU format and supports thin provisioning, snapshots, and copy-on-write operations, making it the standard choice for student VMs. .raw is a flat binary format with higher runtime performance but no native snapshot support — it is typically used when snapshot history is not required and raw speed is preferred.
The environment is segmented into multiple networks, including classroom specific environments for use in FOG deployments. Students in a given course work within an isolated network segment that is separate from other courses and from the management infrastructure. This segmentation prevents cross-class interference and contains any network activity a student generates to their own environment.
Network segments are managed through Proxmox's SDN (Software Defined Networking) layer. Each class is assigned one or more VNets — virtual networks mapped to Linux bridges on the Proxmox hosts. VNets are configurable at the datacenter level in the Proxmox web interface. Proxmox's networking relies on the Linux network stack, giving it flexibility for configurations like NAT (masquerading) and NIC teaming (link aggregation), though fine-grained adjustments require CLI work rather than purely GUI-based configuration.
Access to the environment from off campus requires an OpenVPN connection to the campus network. On campus, The environment is also accessible from dedicated lab computers.
Five domain controller instances are distributed across the cluster to provide redundant identity services. All FSMO (Flexible Single Master Operations) roles — which govern specific AD operations like user account creation, domain naming, and schema changes — are held by dc3-new.cyber.local on PVE14. DC2 and DC3 provide redundant DHCP; the remaining controllers provide DNS. Group policies and user data replicate automatically between all domain controllers.
Note that dc5 (PVE1) and dc2 (PVE8) are currently inaccessible as their host nodes are non-operational. This leaves three active domain controllers. Should both PVE14 and PVE16 go offline simultaneously, DHCP would become non-functional until one is restored.
| VM Name | Services |
|---|---|
| dc5.cyber.local | DNS, DHCP |
| dc4.cyber.local | DNS |
| dc2.cyber.local | DNS, DHCP, CA (Subordinate) |
| dc1.cyber.local | DNS, DHCP |
| dc3-new.cyber.local | DNS, CA (Primary), FSMO |
All student VMs are deployed from pre-built, quality-controlled base templates stored on the cluster. A base template is a VM that has been fully configured, tested, and then converted into a read-only master image. When a course section is deployed, student VMs are created from these templates, giving each student an identical, clean starting environment. This approach avoids the overhead of reinstalling operating systems for every deployment and ensures consistency across all instances of the same course.
Each course has different VM requirements — different operating systems, different network adapter types, and different numbers of VMs per student. These configurations are defined in a central course configuration file. At the start of a semester, student VMs are provisioned based on the enrolled roster and assigned to each student's account in Proxmox, with permissions scoped so that students can only access their own machines.
During midterm periods, student access to their existing VMs is revoked and a fresh environment is provisioned, preventing students from carrying forward prior work into assessment scenarios where that would be inappropriate.
Moving VMs from vSphere to Proxmox is not a native process and requires manual export and import steps. The standard approach is an OVF (Open Virtualization Format) export from vCenter, which produces a portable set of files describing the VM's hardware configuration and disk contents. Those files are then transferred to the Proxmox node via SCP and imported using Proxmox's qm importovf command.
After import, the VM's boot configuration must be corrected manually — setting the disk as the boot device and removing any leftover SCSI references from the vSphere configuration. VMware-specific drivers inside the guest OS are replaced with VirtIO drivers, which are the paravirtualized drivers Proxmox expects. For Windows VMs, post-import checks also verify Active Directory health, DNS resolution, and NTP synchronization to confirm the domain controller is functioning correctly in its new environment.
Not every VM warrants a full disk transfer. VMs with complex configuration state — particularly domain controllers — are worth migrating. VMs built from simple, repeatable processes such as standard Linux servers, Windows workstations, firewalls with exportable configs, and honeypots are better rebuilt from scratch on Proxmox, which avoids driver compatibility issues and results in a cleaner image.
Moving VMs between nodes within Proxmox presents its own challenge: because each node's storage is local by default, Proxmox's built-in migration tools cannot transfer VMs between nodes without shared storage. The workaround is to manually SCP the VM's disk directory from the source node to the destination node and then re-attach the disk to a new VM shell on the target. Once NFS shared storage is fully implemented, live migration between nodes will become available without this manual process.
| Issue | Notes |
|---|---|
| "You have not turned on protection against thin pools running out of space" | Can be ignored. Per Proxmox staff, the suggested setting does not meaningfully help in practice. |
| "Sum of all thin volume sizes exceeds the size of thin pool pve/data" | The thin pool is over-provisioned. Not immediately harmful while free space exists, but the pool will fail as disks fill. Storage utilization should be monitored proactively. |
| Error 599: Too many Redirections | Reboot the affected machine. Typically caused by the Proxmox host being overloaded during OVF import operations. |
| VM boot loop after VMDK import | The boot order needs to be corrected via qm set commands: assign the imported disk to ide0, set the boot order to ide0, set the bootdisk to ide0, and remove the scsi0 reference. The VM must be stopped before making these changes. |
| PVE1, PVE8, PVE21 non-operational | These nodes were lost during initial cluster setup. DC5 and DC2, which resided on PVE1 and PVE8 respectively, are currently inaccessible as a result. |