v2.9.1

Fixed

• Every incremental-mode backup in 2.9.0 destroyed the checkpoint it had just created, so no chain could form. 2.9.0 stopped counting the backup currently being written as leftover data — correct in itself, but it did so by hiding that directory from every scan. Checkpoint reconciliation runs at the end of a backup, while the backup is still registered as in progress, so it could no longer see the manifest written moments earlier and deleted the brand-new checkpoint as unreferenced. The backup then recorded a manifest naming a checkpoint that no longer existed: the history showed it as "Chainable", the degraded-runs alert was blinded because it trusts that same field, and the next run found the checkpoint missing and took another full backup. Only the display suppression is applied now, and never to a directory that holds a readable manifest.

• A backup now confirms its checkpoint exists rather than assuming it. The copy reported a checkpoint as created whenever one had been asked for, so anything that removed it afterwards went unnoticed and the manifest advertised a backup nothing could chain onto. libvirt is asked directly, and a missing checkpoint is recorded as a note on the backup.

  If you ran 2.9.0, any backup it took in incremental mode carries a checkpoint name that is not in libvirt. Nothing is lost — those backups restore normally — but the next run will take a full backup and lay down a working checkpoint.

v2.9.0

Changed

• "Incremental not possible — the previous backup did not record a checkpoint" now explains itself. Switching a schedule to incremental made the very next run report that, which read as a fault in a full backup that was perfectly good. The message named a symptom and stopped: it never said that only a backup taken with incremental backups switched on records a checkpoint, and never said that the run writing the message had just laid one down, so the next run would be a true incremental. The wording now says both — and when a checkpoint genuinely could not be created, it says that instead, along with which copy method it fell back to.

• The Incremental button used to give advice that could not work. It was disabled whenever no checkpoint existed, with a tooltip saying "take a full backup first" — but a full backup taken from the button beside it records no checkpoint either, so following the instruction literally could never enable it. The only way out was a scheduled run. The button is now enabled whenever the host supports incrementals: with no checkpoint to chain onto, it takes a full backup and records one, which is what the engine was always designed to do.

• The schedule dialog now warns before the fact. Switching backup type to incremental says that the first run will still be a full backup and why. The list of reasons a run can fall back to a full also gained the missing checkpoint — the one cause that is near-certain the moment the setting is saved, and the only one it did not mention.

• Storage Used counts backup directories that nothing can restore from. They were filtered out of every listing, so a volume holding 95 GB could report 40 GB with nothing in the product to explain the difference.

Added

• An alert when a schedule set to incremental keeps producing full backups. If the libvirt push-mode tier fails every night — an AppArmor denial, a stale job, a permissions problem — every run still succeeds as a full backup, so the schedule reported success, the banner still said "Incremental", and the only trace was a note buried in one row. Two consecutive non-chaining runs now raise a warning that quotes the underlying cause.

• An alert for backup directories holding data nothing can use, and an automatic sweep of the ones left behind by an interrupted run once they are a day old. A backend killed mid-copy never reached its own cleanup, so a part-written multi-GB image stayed on the volume forever, invisible to every total the product showed. Retention and the sweep now also run every twelve hours rather than only at startup — a host that simply stays up never reclaimed anything a per-VM pass did not cover.

• Backups that carry a checkpoint are marked "Chainable" in the history list, so it is visible which backup the next incremental will build on.

Fixed

• Opening the schedule dialog too quickly could silently downgrade an incremental schedule to "always full". The dialog reads its initial values once, when it opens, and the button that opens it was clickable before the schedule had loaded. Clicking in that window built the form from blank defaults and then saved them explicitly — overwriting the VM's backup type, frequency, time and retention, with a "Schedule saved" toast and nothing to suggest anything had been lost. The button now waits for the data, and the dialog re-reads it if it arrives late.

• An unreadable backup-schedules.json deleted every schedule on the host. Any read failure — a truncated file, a syntax error, a permissions problem — was treated as "no schedules configured". Every automatic backup then stopped without raising anything (the overdue alert reads the same empty map), and the next schedule saved through the UI persisted that emptiness over the top. Reads now distinguish a genuinely absent file from a broken one, and writes go through a temporary file and a rename so a crash mid-write cannot truncate the store in the first place.

• Two VMs due at the same minute could lose each other's schedule updates. Saving a schedule is a read-modify-write of one shared file and the scheduler fires every due VM at once — and the dialog pre-fills 02:00, so several schedules really do come due in the same tick. Writes are now serialised. This also closes the window where an operator's change, saved while a scheduled run was recording its outcome, was silently reverted with a 200 OK.

• Opening a failed schedule and pressing Save cleared the "last scheduled backup failed" alert. The record of how the last run ended was rebuilt from the request body, which does not carry it — so investigating a failure was enough to hide it until the next run failed, up to a month later on a monthly schedule.

• A backup that had already succeeded could be deleted by the housekeeping that runs after it. The retention pass sat inside the same error handler that cleans up a failed copy, whose first act is to remove the backup directory. An unreadable settings file or a busy directory during that pass therefore destroyed a complete, restorable backup — and swept its checkpoint on the way out, so the next run degraded to a full as well.

• One unreadable manifest could delete an entire backup chain. A manifest that failed to read was indistinguishable from a backup that did not exist, which made an incremental's parent look like a childless leaf; retention removed it, and the peel loop cascaded down the chain, leaving every backup above it permanently unrestorable once the read recovered. Retention and checkpoint reconciliation now refuse to prune anything at all while any manifest is unreadable, and say which one.

• A single unlucky capability probe disabled incremental backups until the service was restarted. virsh domcapabilities reporting no backup support was cached for the process lifetime on the grounds that it could not change — but upgrading qemu and restarting libvirtd changes it. A negative answer is now re-probed; a positive one is still cached permanently.

• Taking a full backup by hand on an incremental-mode VM now asks first. It records no checkpoint, so the next scheduled run had nothing to chain onto and took a full backup as well — with no warning, and different behaviour depending on which screen the button was pressed from.

Upgrade notes

• The v2.7.0 upgrade note claiming that "the first backup taken after upgrading also records a checkpoint" was wrong, and has been wrong since that release shipped. A checkpoint is only ever recorded for a VM whose backups are set to incremental. The note has been corrected in place.

v2.8.2

Fixed

• The in-dashboard upgrade cut its own output off after a few seconds and then looked like it had hung. The endpoint watched the upgrade's systemd unit with systemctl is-active, which exits non-zero for any state other than exactly active — and the upgrade runs as a oneshot unit, which is activating for its entire life. The very first poll therefore read "not running", declared the upgrade finished about two seconds in, killed the live log part-way through npm ci, and reported the exit status of a process that hadn't exited (0).

  The dashboard then moved straight to "Backend restarted. Waiting for new version…" while the build had not yet started, and gave up 90 seconds later — often before the upgrade it was waiting for had actually finished. Both symptoms were the same bug: the upgrade itself was running normally the whole time.

  The unit's state is now read with systemctl show -p ActiveState, which reports the state instead of encoding it in an exit code, and the dashboard waits longer now that it starts waiting at the right moment.

v2.8.1

Fixed

• The in-dashboard update could stop dead after "Pulling latest changes…" with nothing to show for it. Before resetting the working tree, update.sh switched it onto origin/main — and that switch refuses whenever a tracked file has been modified locally, which is exactly the drift the reset immediately afterwards exists to clear. npm install rewriting package-lock.json is enough to cause it. The switch is now forced, matching what the surrounding code already intended.

  The failure was also invisible: the command's output was discarded and the script exited on it without a word, so the update simply stopped mid-stream. It now reports what went wrong and where to look.

v2.8.0

Added

• Backups can now be verified, not just taken. A backup was previously only ever read on the day you needed it — and an incremental is only as good as every backup beneath it. Verify walks the whole chain, confirms each image is present, and runs qemu-img check over every one of them. The result is kept with the backup, so the list shows what was last proved good and when.

• Backups that stop working now say so. A failed scheduled run used to leave nothing but a line in journalctl. The outcome is recorded on the schedule itself, and the Backups page carries a banner for anything that needs attention: a scheduled run that failed, one that is overdue and hasn't fired, a chain with a missing link, and a backup whose last verification failed.

• Metrics go back further than a day. Completed hours are folded into an hourly rollup and kept for 90 days, so 7d and 30d ranges join Live/1h/24h on both the dashboard and each VM's Metrics tab. Raw 30-second samples are still what the short ranges read; they are simply no longer the only thing stored, which is what capped history at 24 hours.

• A test suite and CI. 157 unit tests over the logic that has actually broken before — input validation, the SSRF guard, domain-XML generation and escaping, firewall rule validation, backup scheduling, backup-chain resolution and the clone/restore XML rewriting. CI additionally runs shellcheck over the installer scripts at default severity (the bug that had install.sh executing fragments of its own comments as root is only reported at "style", so a warning-level gate would have missed it), and checks that the four places a version has to be bumped actually agree.

Fixed

• The self-upgrade can finally carry its own migrations. update.sh runs from the copy already on disk — the previous release's — so a release could never ship a fix that needed to run during its own upgrade. That is what made 2.6.0 able to lock operators out of their own dashboards. Once the working tree has moved, the update now hands off to the update.sh it just pulled and lets that finish the job.

  This release is still upgraded by the old script, since that is the one on disk; from the next release onward a fix can travel with the upgrade that needs it.

• Cloning or restoring into a new VM carried over the source's cached backing-file chain. libvirt records that chain in the VM definition, and the copy's disks are standalone — so the new VM described a chain its files did not have. The same stale-cache condition makes qemu refuse to open a disk outright. Found by the new tests.

• Invalid firewall rules now name which rule is at fault, rather than reporting the bad value with no indication of where it came from.

v2.7.1

Fixed

• git clean inside the install directory could permanently break the SSH tab for every existing VM. The service account's home is the install directory itself, so the SSH keypair VirtPilot injects into every VM it builds lives at /usr/local/virtpilot/.ssh/. That path was untracked but not ignored, so a plain git clean -fd — the obvious thing to reach for when a clone gets wedged — deleted it. The backend then quietly generates a fresh keypair, so newly created VMs keep working while every VM created before that moment still has the old key in its authorized_keys and can never be reached from the SSH tab again. The serial console is the only way back in.

  Nothing VirtPilot ships has ever run git clean — update.sh uses git reset --hard, which leaves the directory alone — so no install can have hit this on its own. The keypair is now git-ignored, which is enough for git clean -fd to spare it.

v2.7.0

Backups become incremental, restores become whole-machine, and VMs can be cloned.

Added

• Incremental backups. A backup can now copy only the blocks that changed since the previous one, using libvirt checkpoints and qemu's persistent dirty bitmaps. Each incremental is a qcow2 whose backing file points at its parent, so a chain reads as one complete image — restoring any link rebuilds the whole thing automatically, and the Backups page shows both the link's own size and what the full chain costs.

  Schedules gain a backup type: always full, or incremental with a fresh full every N backups. Capping the chain is deliberate — a long chain is slow to restore and puts every backup after a damaged link at risk. A scheduled run that cannot produce an incremental (the VM was off, the previous checkpoint is gone, the host does not support it) takes a full backup instead of being skipped, and records why on the backup.

  Deleting a backup that later incrementals are built on is refused rather than silently orphaning them; the dialog offers to remove the whole subtree instead. Retention follows the same rule — an expired full backup with a live incremental on top of it survives until that incremental expires too.

• Restore into a new VM. Restoring no longer only means overwriting the VM the backup came from. The backup already stored the domain definition, NICs and cloud-init metadata, so it can now rebuild the entire machine: new UUID, new MAC addresses, a fresh IP where the original had a static one, its own NVRAM, and a regenerated cloud-init seed so the guest boots with the new name as its hostname. This works even when the original VM has been deleted.

• Clone a VM. One action produces a complete, independent copy — new identity throughout, disks copied in full rather than layered on the original. A running VM does not have to be shut down first: the copy is taken through the same point-in-time machinery as a backup.

Changed

• Backing up a running VM no longer freezes its filesystems for the length of the copy. The guest agent was asked to freeze before the copy started and thaw only once every disk had finished — up to an hour of stalled I/O on a large disk, long enough to trip guest watchdogs and hang every application on the machine.

  Where the host supports it, backups now use libvirt's push-mode backup API: qemu takes the point-in-time internally and copies in the background, so the freeze lasts only as long as it takes the job to start. Where it doesn't, the VM is pivoted onto a throwaway overlay for the duration of the copy and committed back afterwards. Both are measured in milliseconds of guest impact rather than minutes, and both are recorded on the backup so you can see which ran.

  The old behaviour is still available as a last resort if neither mechanism works, but it now copies the live disk without a long freeze and marks the result crash-consistent, rather than stalling the guest to reach the same outcome.

• The backup manifest no longer stores the guest password in plain text. It is encrypted with the same key as vm-metadata.json. Manifests written by earlier versions are read as before.

• README no longer claims incremental backups it did not have. It does now.

Fixed

• A port forward could silently punch a hole through a deny-all firewall. The per-VM firewall and port forwarding both wrote to the same iptables chain, and whichever was configured last ended up on top — so setting the firewall first and adding a forward afterwards left the forward bypassing the policy entirely, while the opposite order blocked it. Identical settings, opposite security outcomes. The firewall is now always evaluated first: a forwarded port on a VM with a deny-all inbound policy needs an explicit allow rule, which is visible and fixable.

• Port forwarding never actually worked on a managed NAT network. The DNAT rule was right, but the FORWARD ... ACCEPT that lets the translated packet through was appended below libvirt's own rules — and libvirt's chain ends by rejecting forwarded traffic to the guest bridge. Connections were refused before VirtPilot's rule was ever consulted, while the DNAT counter ticked up as if all was well. The rule is now inserted ahead of libvirt's.

• A snapshot taken while the VM was shut down could never be deleted. libvirt only hands an overlay to the qemu user when there is a running qemu to hand it to, so a snapshot of a stopped VM left a file the service account could neither read nor change permissions on. Deleting it failed with "Permission denied", and the VM was left running off an overlay it could not get rid of. Overlays are now created by VirtPilot itself with the right ownership.

• Applying a deny-all firewall could lock you out of your own firewall. Rules are applied against the VM's IP address, which was resolved from the host's ARP cache — and a deny-all inbound policy is precisely what stops that resolving. The next apply, including the one undoing the policy, failed with "Could not resolve VM IP". DHCP leases are now consulted too, which the guest's reachability cannot affect.

• Restoring a backup could leave the VM unable to start. libvirt caches each disk's backing-file chain in the VM's definition; replacing the image out-of-band left that cache describing a chain the file no longer had, and qemu then refused to open the disk at all. Restore now clears the cache so libvirt re-reads it from the images.

• Invalid firewall rules were accepted and only failed later. A rule with an out-of-range port saved successfully and blew up at apply time, after you had left the page. Rule sets are validated when saved, and rejected with a message naming the rule at fault.

• install.sh ran fragments of its own comments as root. Two comment lines inside the systemd-unit template contained backticks, which the shell executed while writing the file — printing three confusing errors on every install and blanking those comments out of the installed unit.

• Deleting a VM without deleting its storage could be refused once it had been backed up. libvirt will not undefine a domain that still carries checkpoint metadata unless told to drop it, and nothing else clears it when the disks are being kept. virsh undefine is now asked to remove it, with a fallback for libvirt older than 5.6.

• Un-ticking a storage directory's "use as default" checkbox appeared to work but never could — every purpose must always have exactly one default. The checkbox is now disabled once the directory holds the flag, with an explanation.

• Malformed request bodies returned "Internal server error" instead of a 400, and filled the log with stack traces. Several operations that were refused for legible reasons — restoring onto a running VM, snapshotting a VM with backup checkpoints, deleting a template that does not exist — reported themselves as server faults rather than 4xx.

• A restore refused for a reason the operator can act on — the VM is still running, the name is taken, the backup's chain is incomplete — now returns the matching 4xx status instead of reporting a server error.

Upgrade notes

• Existing backups keep working untouched and are treated as full backups. The first backup taken after upgrading also records a checkpoint, which is what makes the next one able to be incremental — so incremental backups become available one run after the upgrade, not immediately.
• Incremental backups require a running VM and a host whose virsh domcapabilities reports <backup supported='yes'/> (libvirt 6.0+ with a qemu that has blockdev-backup — anything on Ubuntu 22.04 or newer). The Backups page says which of these is missing when the Incremental button is unavailable.
• Cloning and restore-into-a-new-VM carry over attached ISOs but not the source's cloud-init seed, and turning off "give the clone its own identity" will duplicate a static IP — the dialog says so.

v2.6.1

Fixed

• Upgrading to 2.6.0 locked operators out of nginx-fronted installs that had an IP allowlist configured. 2.6.0 stopped trusting X-Forwarded-For unless TRUST_PROXY was set, which was the right call for a directly-exposed backend but wrong for the standard install: every request then resolved to 127.0.0.1, matched no allowlist entry, and login returned 403 IP not allowed while the service itself reported healthy.

  The TRUST_PROXY migration shipped in 2.6.0's update.sh could never have prevented this. The in-dashboard self-upgrade executes the update.sh already on disk — the previous version's — so a migration only takes effect from the release after the one that needs it.

  The bind address now settles it without an env var. A backend bound to loopback is unreachable from the network, so the only thing that can set X-Forwarded-For is a proxy on the same host, and one hop is trusted. A backend bound to a routable address may be directly exposed, so it keeps ignoring the header until TRUST_PROXY says otherwise. An explicit TRUST_PROXY still wins in both cases, so existing configurations are unaffected.

  If you were locked out by 2.6.0, this release restores access on its own — no .env edit needed. Adding TRUST_PROXY=1 by hand remains a valid fix and is left alone if present.

v2.6.0

Security and robustness release from a full audit of the codebase. Two changes need operator attention — see Upgrade notes at the end.

Security

• Proxy headers are no longer trusted by default. The backend trusted one proxy hop unconditionally, so any install reachable directly (BIND_ADDRESS=0.0.0.0 without a reverse proxy) let an unauthenticated client spoof X-Forwarded-For and walk straight past login rate limiting, 2FA rate limiting and the IP allowlist. Client IPs now come off the socket unless TRUST_PROXY opts in; update.sh sets it automatically for nginx-fronted installs.
• A failed secret decryption no longer silently disables 2FA and the IP allowlist. getUserSettings() caught every error and returned defaults, so a rotated or mismatched ENCRYPTION_KEY turned two-factor auth off and emptied the allowlist without a word — and the next settings save made it permanent by overwriting the TOTP secret. Only a genuinely absent settings file now yields defaults.
• Disk-download tickets are no longer accepted as full session tokens. The short-lived ticket that authorises a single disk download travels in a URL query string (so it lands in proxy logs and browser history) but satisfied every authenticated endpoint and WebSocket. Session tokens now carry a typ claim that tickets deliberately lack.
• VM creation can no longer inject arbitrary libvirt device XML. nicModel from the create request reached the generated domain XML unvalidated and unescaped, so a crafted value could close the attribute and append devices — an extra <disk> pointing at a host block device, a <serial> writing to an arbitrary path — into a domain that is then persisted by virsh define. All interpolated values are now escaped at the builder, and the route validates before anything touches disk.
• The service account can no longer rewrite the script that runs as root. update.sh was owned by the unprivileged service user while sudoers granted it NOPASSWD root execution. It is now root-owned, and the (unused) grant is gone. The systemctl and apt-get grants have been narrowed to the exact commands the dashboard issues — the previous bare apt-get grant was a root shell in disguise.
• Logging out now survives a restart. Revoked tokens were held in memory only, so a service restart — which every self-upgrade performs — brought them back to life. Revocations are now persisted (hashed) in the existing database.
• The download SSRF guard actually resolves hostnames now. It classified only literal addresses, so a hostname pointing at a private or link-local address passed, and IPv4-mapped IPv6 literals ([::ffff:169.254.169.254]) matched none of its patterns. It now resolves the host, rejects any private answer, and pins the connection to the address it validated.
• Snapshot descriptions are passed as a named option instead of a bare argument, so a description starting with -- can no longer be parsed by virsh as a flag (--print-xml made the API report success while creating nothing).
• Uploaded archives are now capped on decompressed size, and the SSH WebSocket's pre-connection input buffer is bounded.

Fixed

• Three ways to kill the backend, one of them unauthenticated. A login attempt with a non-ASCII password crashed the process outright — the length check counted UTF-16 units while the comparison used UTF-8 bytes, and the mismatch threw inside an async handler. An ISO download filename containing a space threw outside its error handler. WebSocket handlers registered their error listener only after several awaits, so a malformed frame arriving in that window took the whole daemon down — along with every live console, SSH and VNC session. All three are fixed, and process-level guards now keep the service up rather than letting one bad request end it.
• Console, SSH and VNC sessions no longer leak. A client that vanished without closing cleanly (laptop lid, dropped Wi-Fi) left a virsh console pty, an SSH channel into the guest, or a VNC socket to QEMU held open forever — and the VNC leak blocked all future VNC sessions for that VM. All three now use a heartbeat, and a client that disconnects mid-connect no longer leaves a guest shell behind.
• Deleting an "orphaned" disk during a libvirt outage no longer deletes live VM disks. When libvirt was unreachable the check for "is this VM still defined?" failed open, the Storage page marked every disk orphaned, and the delete button removed real data. The check now fails closed and returns an error.
• Uploads and snapshot exports no longer overwrite live disk images. Templates are backing files for every VM created from them; writing over one corrupted running guests. Colliding writes now return a conflict, and the cross-filesystem move path writes to a temporary file and renames rather than rewriting an inode QEMU has open.
• Firewall rules are no longer torn down before they are validated. The existing chains were flushed first, so a rule set that failed to apply left the VM with no filtering at all while the API reported success. Rules are now built and validated up front, applied strictly, and rolled back on failure. Stale rules from a previous VM IP are swept properly.
• Port forwarding no longer hijacks other guests' traffic. The NAT rule matched on destination port alone, so forwarding a host port to one VM silently redirected every other guest's outbound traffic on that port. Rules now match on the inbound interface; existing rules are reconciled automatically at startup.
• A backup schedule with an unrecognised frequency no longer loops every minute until the disk fills.
• VM names accepted at creation now match what the rest of the app accepts. The create form was more permissive than the validator, so a name like _build01 created a VM that could not then be opened, started or deleted from the dashboard.
• Cloud-init passwords containing : or \ now work. They were escaped for a format that treats neither character specially, so the backslashes became part of the guest password while the dashboard displayed the original.
• VM metadata writes no longer race. Concurrent updates could interleave and a torn file was read as "no metadata for any VM", which the next save then made permanent. Writes are serialised and atomic.
• A failed bridge-network creation no longer leaks the bridge and wedges the name counter; a failed port-forward no longer leaves an unremovable rule behind.
• uninstall.sh no longer deletes VM disks it did not create — it selected any domain whose XML merely mentioned the storage path, which matched every VM and some hand-made ones.
• install.sh no longer disables nginx's default site, and bootstrap.sh no longer redeploys a stale commit while reporting success.
• Frontend: the starter-template downloader could run twice across tabs and corrupt the download; the Backups page hid backups belonging to deleted VMs (and excluded them from the storage total); the network step of the VM wizard let you continue without a required static IP; snapshot-to-template timed out in the UI while succeeding on the server; a cleared disk-size field silently provisioned 20 GB.

Changed

• Re-running install.sh no longer destroys an existing install. It regenerated JWT_SECRET and ENCRYPTION_KEY and rewrote .env wholesale — orphaning every encrypted value on disk and discarding operator edits. It now reuses existing secrets, keeps the current password unless VP_RESET_PASSWORD=1, and preserves unrelated settings.
• Dependencies updated for published advisories, including tar (critical — used by backup and restore) and ws (high — every console session).

Upgrade notes

• Behind a reverse proxy: update.sh adds TRUST_PROXY=1 automatically when it detects the nginx site. If you front VirtPilot with something else, add TRUST_PROXY=1 to packages/backend/.env yourself, or client IPs will all appear as the proxy's address and an IP allowlist will lock you out.
• Replacing a template or ISO by uploading over an existing one now returns a conflict instead of silently overwriting. The dashboard does not yet offer a "replace" toggle — delete the old file first.
• VMs created with a name the validator rejects (a leading underscore, or longer than 63 characters) remain unusable from the dashboard; rename them with virsh domrename.
• Existing sudoers rules are not rewritten by update.sh — re-run install.sh to pick up the narrowed grants.

v2.5.4

Changed

• Dashboard health band no longer truncates long hostnames. The host status line previously crammed the hostname and VM count onto one line (slc-gateway.hostedai.dev · 1/1 VMs running), so longer hostnames were cut off. The hostname now sits on its own line with the VM count beneath it; the full name is available on hover. The status block stays vertically centred against the vitals and version chip, so card alignment is unchanged.