v0.67.0

0.67.0 - 2026-09-26

Highlights

• Power lab hosts only when work needs them. Static SSH power hooks start a dedicated host before its first lease and shut it down after the last lease, retaining retryable custody if shutdown fails; the explicit dedicated-host contract keeps power control with one controller.
• Use Proxmox through a runtime adapter with safer lease recovery. adapter serve now supports fixed-ID Linux workspaces; VMID reservations skip locally bound IDs and are serialized through durable publication to prevent concurrent collisions. Definite native pre-allocation 401/403 rejections free the attempt, while proxy-generated 403s and other ambiguous replies retain custody.
• Keep forwarded environment values out of command lines. SSH commands, preflight, Windows/WSL2, and cache warming receive values through private temporary files over stdin; unsupported NUL values are rejected by variable name, and checkpoint archives exclude staged environment files left by interrupted cleanup.
• Recover coordinator failures without duplicate allocations or lost leases. Atomic lease admission and publication, exact rereads after ambiguous commits, and provider-owned failure facts preserve recovery authority; Azure leases settle after definite VM rejection and verified companion cleanup. Kept warmed leases also honor recorded idle/TTL expiry where automatic reaping is supported.
• Use Azure GA ephemeral disks and managed identities. Eligible VMs can use ephemeral OS disks with full caching, while managed disks remain the default; direct VMs can attach an existing user-assigned managed identity and verify it before fixed-lease readiness.
• Spend less time preparing sync snapshots. On a shared Apple M3 Ultra, unchanged-fixture median snapshot preparation fell from 5.195 s to 2.698 s for a realistic repository and from 48.547 s to 25.378 s for 20,002 files. The local Git-seed path uses 17 Git calls instead of 50 and reads 25% fewer logical source bytes while retaining independent content verification.
• Run more reliably across local and hosted targets. SSH workspace locks work with BusyBox flock, Blacksmith joins canceled sync children and closes native SSH masters before returning, and WSL2 static targets support non-administrator Windows accounts. Parallels IP discovery stops admitting probes at its deadline and preserves the latest VM-query error instead of giving stale recovery advice.

Upgrade notes

• Replace azure.osDisk: ephemeral-preview, --azure-os-disk ephemeral-preview, and CRABBOX_AZURE_OS_DISK=ephemeral-preview with ephemeral, including coordinator environment settings. GA full caching requires at least 8 active vCPUs, a supported family (N/L/M/H, D/DC/E/Eb/EC v5-v7, or F v6-v7), and sufficient local storage; existing ephemeral settings now use full caching, so smaller VMs and Fsv2 no longer qualify. managed remains the default and the option for smaller VMs or native checkpoints; released CLIs sending the removed value to an updated coordinator receive HTTP 400 before allocation. PR 2565, PR 2567. Thanks @jwmoss.
• Static SSH power hooks require static.power.dedicated: true in trusted user configuration, exclusive host ownership by one controller/state directory, a canonical IP, explicit SSH credentials, and absolute hook executable paths. Preserve the claims and ssh-power state until shutdown completes; use distinct static IDs for concurrent leases, and do not use prepared run --id reuse while power custody remains. PR 2545, PR 2576. Thanks @altaywtf.
• warmup.keep is a new configuration key and still defaults to true; retention across runs no longer exempts kept warmups from recorded idle/TTL expiry where automatic reaping is available. Recreate existing direct GCP guests to install the updated expiry guard, and ensure their service account can delete the VM. Manual cleanup still skips kept machines; for providers without an automatic idle reaper, use warmup.keep: false with scheduled cleanup or stop explicitly. Tenki retains its documented sticky-lease exception. Issue 2536, PR 2553. Thanks @youssef-tharwat for the report.
• Drain Proxmox adapter workspaces under their original configuration before changing the token ID, endpoint, node, or clone settings; rotating only the token secret is supported. For older prepared claims left by definite clone rejection, restore permissions and use checked stop --force recovery only after inspecting the task and VM; keep claims after ambiguous proxy responses. PR 2577, PR 2562, PR 2566, PR 2570. Thanks @ahkohd.
• SSH command environment values containing NUL now fail before upload; remove the NUL or pass binary data as a file instead. Empty and multiline values remain supported. PR 2573.
• Daytona API-key fixed acquisition now requires organizationId from /api-keys/current; older deployments must update that endpoint or use an OAuth organization profile. Existing saved claims remain supported. PR 2574. Thanks @Patrick-Erichsen.
• Ordinary Blacksmith Testbox runs now require macOS/Linux process-group support and compatible ps, matching artifact runs; unsupported controllers fail before acquiring a lease. PR 2547.

Added

• Power dedicated static SSH hosts on for their first lease and off after the last, with durable host references, retryable shutdown, and joined hook cancellation. Issue 2544, PR 2545, PR 2576. Thanks @altaywtf.
• Run Proxmox fixed-ID Linux workspaces behind adapter serve, with immutable routing and token-identity scopes, secret rotation, and exact registered cleanup receipts so deleted VMs do not leave workspaces stuck stopping. Issue 2558, PR 2560, PR 2577. Thanks @ahkohd.
• Attach an existing user-assigned managed identity to direct Azure VMs with azure.userAssignedIdentityResourceId or --azure-user-assigned-identity-resource-id, verifying attachment before fixed-lease readiness or adoption. To add an identity to an existing fixed lease, stop it and allocate a replacement with a new lease ID. PR 2575. Thanks @galiniliev.

Fixes

• Keep allowlisted SSH environment values out of command lines and workspace-owner launchers using private temporary files delivered over stdin, including preflight, Windows/WSL2, and cache warming; reject NUL-containing values by variable name and refuse remote cleanup after workspace ownership renewal fails. Issue 2535, PR 2556, PR 2573. Thanks @youssef-tharwat for the report.
• Exclude staged SSH command environment files left by interrupted cleanup from workspace checkpoint archives. PR 2569.
• Reserve Proxmox VMIDs without colliding with live local bindings or concurrent local acquisitions, checking alternatives against cluster inventory and holding the reservation fence through durable claim publication. Issue 2559, PR 2566, PR 2571. Thanks @ahkohd.
• Retire Proxmox prepared claims after definite native pre-allocation authorization rejection and support checked stop --force recovery for absent, unbound attempts; retain custody after proxy-generated, task-bearing, or otherwise ambiguous 401/403 responses. Issue 2559, PR 2562, PR 2570. Thanks @ahkohd.
• Commit legacy coordinator lease admission and publication atomically and resolve ambiguous commits through exact bound-record rereads, avoiding duplicate provisioning while preserving cancellation and cleanup authority. Issue 1561, PR 2555.
• Retain unresolved coordinator allocations after cancellation without a cloud ID, using provider-owned provisioning-failure facts while core owns recovery timing. Settle Azure leases only after definite VM rejection and verified companion cleanup, preserve uncertainty across mixed fallback failures, and explain scheduled cleanup retries in stop. PR 2478, PR 2552, PR 2578. Thanks @steipete.
• Allow Azure fixed-lease stop --force to finish retained claims after external cleanup, verifying VM and companion absence with read-only checks and preserving interrupted recovery for retry. PR 2572. Thanks @galiniliev.
• Honor recorded idle/TTL expiry for kept direct cloud warmups, including the GCP guest expiry guard, and add warmup.keep without changing the retention default or manual cleanup protection. Issue 2536, PR 2553. Thanks @youssef-tharwat for the report.
• Support BusyBox flock for SSH workspace locks without mistaking its unsupported timed-wait option for workspace contention. PR 2579.
• Finish Blacksmith sync-child cleanup before returning from canceled runs and disable detached native SSH masters for sync and artifact transfers, closing connections after successful, failed, or canceled runs. PR 2547, PR 2580.
• Support WSL2 static targets under non-administrator Windows accounts by removing WMI shell discovery and accepting read-only app-capability grants on HOME while keeping private staging ACLs strict. Issue 2543, PR 2546, PR 2554. Thanks @altaywtf.
• Stop Parallels IP discovery from admitting probes after its startup deadline, preserve caller cancellation and the final VM-query error, and suppress stale DHCP or clone-mode advice when current inventory is unavailable. PR 2480, PR 2548. Thanks @steipete.
• Preserve Lambda acquisition causes when rollback fails, report recovery-claim write failures, and block automatic fresh-allocation retries after failed cleanup. PR 2463. Thanks @steipete.
• Bound Tencent Cloud IP-readiness reads and retry waits, preserve cancellation and acquisition causes after rollback failure, and prevent cleanup errors from triggering fresh-instance retries. PR 2464. Thanks @steipete.
• Keep background telemetry from replacing workspace command witnesses and failing successful hydration reads with exit 74. PR 2563.
• Report missing local claims accurately when replaying stop after verified absence, retaining exit 1 without a terminal receipt and preserving provider resolver and genuine identity-mismatch errors. PR 2551.

Changed

• Use GA full caching for Azure ephemeral OS disks through the SDK and coordinator, with supported-family and active-vCPU eligibility checks; managed OS disks remain the default. PR 2565, PR 2567. Thanks @jwmoss.
• Build local Git-seed snapshot manifests once, validate captured Git inputs at acceptance, reuse copied-file digests between mandatory content reads, and avoid rescanning unrelated parent directories. The unchanged 20,002-file fixture used 17 rather than 50 Git calls and 894,566,442 rather than 1,192,755,256 logical source bytes; median snapshot preparation fell from 48.547 s to 25.378 s on a shared Apple M3 Ultra. These measure local preparation, not network-transfer throughput; final uncached content and byte-count verification remain mandatory. PR 2581.
• Move direct Daytona integration to the maintained Go clients and typed organization/lifetime metadata, and reconcile fixed cleanup through exact sandbox lookup so retirement of the paginated endpoint cannot strand cleanup. PR 2574. Thanks @Patrick-Erichsen.