Proxmox and Cloud Director Developer Guide

Proxmox and Cloud Director — Developer Guide

Since 3.1.0 · PR #167 by Andrew Turbay (@turbay-a) · User guides: Proxmox VE · VMware Cloud Director

How the two hypervisor syncs work, and what changed when the PR was merged, and why. Each change is written as the PR did X, the problem was Y, now it does Z, because most of them guard against something that only shows up on a real install, often as data quietly deleted. The rules are marked in the code with TRAP: comments (Code traps):

git grep -n "TRAP:" -- includes/proxmox_sync.php includes/vcloud_sync.php includes/hypervisor_http.php

---

Thank you, Andrew

This is Andrew's third contribution, after Telegram and Teams and Mattermost, and it shows real operational experience. The rules about what not to delete came from a sync he'd run in production, and they're the right instincts. Kept exactly as he built them:

• Several servers of each, each with its own row, schedule and data, keyed by server: (connection_id, vmid), (connection_id, vm_uuid).
• Read-only. Nothing ever writes to Proxmox or Director.
• Credentials encrypted at rest with encryptValue(), never returned to the browser (has_password only), and a blank or masked field on edit keeps the stored one.
• IPs from the VM's own NICs only. The QEMU guest agent reports docker0, veth… and bridges too; only interfaces whose MAC is one of the VM's configured NICs count.
• An offline node keeps its VMs, and the safety guard: fewer than half of the known VMs seen means nothing is deleted.
• Director's XML read by local element name, so provider and tenant namespaces both parse.
• The data shaped like vCenter's (details_json uses the raw_data shape), so the Servers page's detail view shows Proxmox and Director VMs with no special cases.
• A Source column on Servers, two settings tabs, two capabilities marked sensitive, schema in all three places, foreign keys in Database Verification, and every string translated into 14 languages.

---

Where things are

	File
⚙️	includes/hypervisor_http.php	New at merge. HypervisorHttp: https only, the shared CA bundle, the test hook
⚙️	includes/proxmox_sync.php	Login (token or ticket), the sync, proxmoxSaveConnection(), proxmoxConnectionOut()
⚙️	includes/vcloud_sync.php	Session, paged inventory, NICs and disks, edge gateways, logout, vcdSaveConnection()
🔌	api/assets/proxmox_connections.php, vcloud_connections.php	List, save, delete. assets.proxmox / assets.vcloud
🔌	api/assets/proxmox_sync.php, vcloud_sync.php	Test and Sync now for one server. Same capabilities
🔌	api/assets/proxmox_vms.php, vcloud_vms.php	Show VMs (and edge gateways). Same capabilities
🔌	api/assets/sync_hypervisors.php	New at merge. The Servers page's sync: every active server, on module access
🔌	api/assets/get_servers.php	Servers list: vCenter, Proxmox and Director rows in one shape
⏰	cron/proxmox_sync.php, cron/vcloud_sync.php	Scheduled; CLI only; each server on its own interval
🖥️	asset-management/settings/index.php, assets/js/proxmox-settings.js, vcloud-settings.js	The two settings tabs
🧪	tests/hypervisor-sync.php	New at merge. 48 checks against a stand-in of each API, rolled back
🗄️	proxmox_connections, proxmox_nodes, proxmox_vms, vcloud_connections, vcloud_vms, vcloud_edge_gateways	Install-wide, like vCenter's servers

---

1. Every request goes through one door

HypervisorHttp::request() is the only cURL call in either sync.

HTTPS only. The PR's address check was #^https?://#, so http://pve:8006 saved and synced. The problem: every Proxmox request carries the API token secret in a header, and every Director login carries the password in Basic auth, so plain http hands them to anyone on the path. The error message even said "must start with https://"; the regex disagreed. Now HypervisorHttp::validateHost() refuses http:// when a server is saved, and request() refuses it again, in case a row got one some other way.

The CA bundle. The PR set CURLOPT_SSL_VERIFYPEER from the server's "Verify certificate" tick, but no CURLOPT_CAINFO. On WAMP, and on some Linux and Docker hosts, PHP has no default bundle, so a perfectly valid certificate fails with "unable to get local issuer certificate". That's GH #129 again, which sslApplyCurl() exists to prevent. Now a ticked box calls sslApplyCurl($ch, true): verification forced on, with the install's bundle, like every other integration. An unticked box turns verification off for that server only, which Proxmox's default self-signed certificate usually needs.

The test hook. HypervisorHttp::$testTransport lets the test answer as Proxmox or Director. Same idea as MessagingProvider::$testTransport and TeamsProvider::$testKeys; never set outside tests/.

2. Proxmox: what gets deleted

// includes/proxmox_sync.php — the lists that came back
$list = proxmoxTryGet($c, $auth, '/nodes/' . rawurlencode($nodeName) . '/' . $segment);
if (!is_array($list)) {
    $summary['skipped_nodes'][] = $nodeName . '/' . $segment;
    continue;
}
$readLists[$nodeName . '|' . $type] = true;

The PR removed a VM when it was missing and its node "answered", meaning it was online in /nodes. The problem: a node can be online while its qemu or lxc list fails (a timeout, a permission on that node). That node still counted as scanned, so every VM on it looked gone and was deleted. The safety guard didn't catch it either: it compared VMs seen across the whole cluster with the known ones, so one small node failing in a big cluster stayed under the 50% line. Now removal is scoped to (node, type) lists that came back, and the guard compares, within those lists, how many known VMs are still there. Rows are upserted before the removal looks, so a VM that moved node is already recorded on its new one.

Proven by a control: with the PR's rule put back, tests/hypervisor-sync.php fails "a node whose qemu list failed keeps its VMs", with 2 VMs deleted that still existed.

Test and the node list. Test read the node list with proxmoxTryGet(), which swallows errors. With a token there's no separate login call, so the node list is the first request, and an unreachable server came back as "Logged in, but the node list could not be read. Check the user's permissions". That's wrong, and it sends an admin into Proxmox permissions for a typo in the address. Now Test and the sync read the node list with proxmoxCall(), and the real reason ("Could not reach the Proxmox server: Could not resolve host…") is what the admin sees. Found by testing the real endpoint with a server that doesn't exist.

3. Director: what gets deleted

Paging. The PR stopped reading the VM list when $page * 128 >= total, assuming each page held the 128 it asked for. The problem: Director caps a page at its own maximum (restapi.queryservice.maxPageSize, which an administrator can lower). On such a system page 1 came back short, the loop stopped, the list counted as complete, and every VM after page 1 was deleted, on every sync. Now the loop counts the records that actually arrived and stops at total. The stand-in pages by 4 while being asked for 128, which is what found this.

Edge gateways. The PR read one page of 128 gateways and then deleted every gateway not on it. Over 128 gateways, the rest were deleted each sync; and a failed read with a partial list did the same. Now every page is read (pageCount), and gateways are removed only after a complete read.

Logging out. The PR logged in on every Test and every sync and never logged out, so each scheduled run left a live session on Director until it timed out: one more per server per interval. Now vcdLogout() deletes the session after Test, after a sync, and after a sync that failed part-way.

A colon in the user name is refused when saving: Basic auth splits user@org:password at the first colon.

4. The Servers page

The PR's Sync button fetched the server lists from proxmox_connections.php and vcloud_connections.php and synced each one from the browser. The problem: those endpoints need the settings permissions, so for anyone who only uses the Servers page they returned 403. Both sources were skipped silently, and the button reported success. vCenter had settled this question: get_vcenter.php syncs on module access, because the Servers page is operational and a sync changes no configuration. Now sync_hypervisors.php does the same for every active Proxmox and Director server in one call. Adding, changing, testing and "Show VMs" still need the capabilities.

Every sync on an install without vCenter showed a red "vCenter settings not configured". get_vcenter.php now says not_configured: true, and the page skips that line when other sources exist.

The Source badge used fixed colours; it now uses the theme tokens (--accent-soft, --surface-2…) so it works in dark mode, as do the status badges in both settings tabs. The empty table said Click "Sync vCenter"; a new string names the new button. (It's a new key rather than a changed one, so the existing translations don't drift.)

5. Smaller things

• Director's Show VMs printed "undefined". The table was copied from Proxmox's and still read vmid, vm_type and node_name, which Director VMs don't have. It now shows Name, vApp and Organization VDC, and the edge gateways, which were synced and returned but never displayed.
• A foreign key named twice. freeitsm.sql called one key fk_vcloud_edge_connection; Database Verification looks for fk_vcloud_edge_gateways_connection. On every fresh install it wouldn't find it and would add a second, identical key. The schema now uses Verification's name.
• Deleting a server also deletes its rows explicitly, not only through the cascade. Verification adds the foreign keys to an existing install only when it runs.
• Saving moved into the includes (proxmoxSaveConnection(), vcdSaveConnection()), so the rules are in one place and the test runs the real ones. Editing a server that doesn't exist is refused.
• Help. The PR's two sections were English written into the page, between sections 1 and 2, with no number and no menu entry. They're now sections 11 and 12, after Servers, in lang/en/asset-management.php (help.proxmox.*, help.vcloud.*), with the full guides here on the wiki instead of docs/. The menu had two entries numbered 15; it now counts 1 to 19. Two menu items printed & because their strings already held &; they're echoed without double escaping.
• Proxmox VE 9 removed VM.Monitor. Guest-agent reads now need VM.GuestAgent.Audit. The docs give both.

---

What a sync does, in order

Proxmox: log in (token header PVEAPIToken=user@realm!id=secret, or a ticket from /access/ticket sent as the PVEAuthCookie) → /nodes → /cluster/status for the name → for each online node, qemu and lxc lists → each VM's config, and for a running VM the guest agent's interfaces → upsert → remove what a list that came back no longer has, unless the guard fires → record the result on the server row.

Director: POST /cloudapi/1.0.0/sessions with Basic user@org:password; the token comes back in X-VMWARE-VCLOUD-ACCESS-TOKEN → /api/query?type=vm page by page → for each VM, networkConnectionSection and virtualHardwareSection/disks (ResourceType 17 is a disk) → upsert → remove only after a complete read, unless the guard fires → edge gateways, every page → log out → record the result.

The test

tests/hypervisor-sync.php: 48 checks, everything in one transaction that's rolled back. The stand-ins hold a small cluster and a small Director in arrays, with switches to make one call fail. What it proves: https only; the secret encrypted, never returned, kept on a blank or masked edit; both Proxmox logins; IPs from the VM's own NICs; container ip= and DHCP; a failed qemu list keeps that node's VMs while its lxc list still applies; offline nodes; the guard; Director paging past its own page cap; a failed page removes nothing; edge gateways across pages; logout; an unreachable server named as such.

Not run against a real server. Proxmox was run against a real cluster by the contributor before the merge changes; Director has never met a real one. The first live run of each is worth watching.

---

See also: Proxmox VE · VMware Cloud Director · Code traps · Assets