feat(cluster): worker-as-LXC architecture (Linux v1) - #316
Conversation
Adds host_lan_ip collision check (409 on duplicate) in register_worker, find_worker_by_host_lan_ip to ClusterManager, degraded/degraded_reason fields to WorkerInfo, and privilege+nesting verification in incus_enroll.
Adds PHASE variable (host/inside) driven by TAOS_INSIDE_WORKER env var, and refuse_flat_mode_install() which blocks re-runs on hosts that already have flat-mode taos-agent-* LXC containers.
…, worker LXC) Adds check_kernel_features (cgroup v2, btrfs, nftables), worker_disk_cap with TAOS_WORKER_DISK_CAP override, create_btrfs_loopback for incus storage, launch_worker_lxc with privileged+nesting config, and phase1_host_prep which wires them together with per-distro package install.
Adds setup_port_forward (nft dnat rule host:8443 -> worker LXC:8443, persisted to /etc/nftables.conf), reexec_into_worker_lxc stub, and the phase 1 entry-point block that runs on bare Linux hosts, calls phase1_host_prep + setup_port_forward, then exits (T8 will replace the exit 0 with the re-exec call).
… loopback incus storage create with source= on a pre-formatted image fails on ext4 hosts (incus 6.x checks that the source path resides on btrfs). Switch to the size= parameter so incus manages the loopback image itself, which works on any host filesystem.
The LXC exec-reachability check in launch_worker_lxc passes before the container's DHCP address is assigned. Add a 15-iteration / 30-second retry loop so setup_port_forward waits for the IP rather than dying immediately.
incus storage list does not support -c flag; use plain --format=csv and awk to extract the name column. Also fix incus list -c name (which expands to multiple columns) to -c n for container name-only output.
- Add phase2_inside_lxc(): apt-installs incus+bees, inits nested incus, writes bees.service unit (TAOS_NO_DEDUP opt-out), detects host LAN IP via default route gateway, then calls install_and_enroll_incus(). - Wire phase 1 entry-point to call reexec_into_worker_lxc() instead of the T7 placeholder log+exit; add phase 2 (inside) entry-point block. - Add worker registration POST to /api/cluster/workers inside install_and_enroll_incus() (step 7) with host_lan_ip and worker_lxc_image_version fields before /incus-enroll (step 8). - Make sg_incus wrapper root-aware: skip incus-admin group ceremony when running as root inside the privileged worker LXC.
… init When run via 'curl | bash', stdin is the curl pipe. incus launch and incus admin init read from stdin for YAML config; they slurp the rest of the script, causing 'yaml: unmarshal errors' failures. Fix by redirecting stdin to /dev/null on all incus commands that may read it.
…04 repos) bees (btrfs dedup daemon) is not packaged in Ubuntu 24.04 Noble. Install with apt-get and skip the bees.service setup gracefully if not available, rather than failing the entire Phase 2 run.
…ntry-points phase2_inside_lxc() calls install_and_enroll_incus(), but that function was defined after the phase if-blocks, so it was not yet registered when the inside-phase exit path ran. Move the definition up (before the phase entry-point blocks) so it is always available. Remove the duplicate definition that remains in the flat-install section.
Adds tinyagentos/cli/worker.py with three subcommands (convert-to-lxc, dedup, resize-storage), registers taos-worker-ctl console script in pyproject.toml, and covers all subcommand parsers with 11 unit tests.
|
Warning Rate limit exceeded
To keep reviews running without waiting, you can enable usage-based add-on for your organization. This allows additional reviews beyond the hourly cap. Account admins can enable it under billing. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (22)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![kilo-code-bot[bot]](https://avatars.githubusercontent.com/in/2193792?s=60&v=4){kind=small}
| export TAOS_INSIDE_WORKER=1 | ||
| export TAOS_CONTROLLER_URL='${CONTROLLER_URL}' | ||
| export TAOS_WORKER_NAME='${WORKER_NAME}' | ||
| curl -sL '${REPO}/raw/${BRANCH}/scripts/install-worker.sh' | bash -s -- '${CONTROLLER_URL}' |
There was a problem hiding this comment.
CRITICAL: Downloading and executing script from remote URL without integrity verification poses a security risk. Consider adding GPG verification or checksum validation to prevent MITM attacks or tampered scripts.
| subprocess.run(["sudo", "incus", "stop", "taos-worker"], check=True) | ||
|
|
||
| print(f"Resizing {pool_img} to {new_size}...") | ||
| subprocess.run(["sudo", "truncate", "-s", new_size, pool_img], check=True) |
There was a problem hiding this comment.
WARNING: Truncating the storage pool image file as root without validation of the new size could lead to data loss if the size is incorrect or smaller than current usage. Add checks to ensure the new size is larger than current used space.
| die "unsupported package manager" | ||
| fi | ||
| sudo systemctl enable --now incus | ||
| sudo incus admin init --minimal < /dev/null 2>/dev/null || true |
There was a problem hiding this comment.
WARNING: Suppressing errors from 'incus admin init' with '|| true' may hide failures that could cause issues in later steps. Consider handling specific error codes or logging warnings instead.
| if command -v apt >/dev/null 2>&1; then | ||
| sudo apt update -y | ||
| sudo apt install -y incus nftables curl btrfs-progs | ||
| elif command -v dnf >/dev/null 2>&1; then |
There was a problem hiding this comment.
SUGGESTION: Persisting nftables ruleset by overwriting /etc/nftables.conf directly may not be atomic if the command fails midway. Consider writing to a temporary file first and moving it.
Code Review SummaryStatus: 1 Issue Found | Recommendation: Address before merge Overview
Issue Details (click to expand)CRITICAL
Other Observations (not in diff)Issues found in unchanged code that cannot receive inline comments:
Files Reviewed (22 files)
Fix these issues in Kilo Cloud Reviewed by grok-code-fast-1:optimized:free · 266,212 tokens |
…convert + quota T11: test_worker_lxc_install — fresh ubuntu VM, install-worker.sh, assert LXC config (privileged+nesting), nftables :8443, nested incus, controller registration. T12: test_convert_flat_to_lxc — legacy flat-mode install, stub agent deploy, memory dir hash before/after taos-worker-ctl convert-to-lxc, assert data survives. T13: test_disk_quota_enforcement — 1 GiB quota fill + ENOSPC assertion, agent isolation check, bees dedup bytes_deduped_total assertion (auto-skips if bees absent). All tests guarded by @pytest.mark.skipif(not INTEGRATION, ...) — collect cleanly in CI without TAOS_INTEGRATION=1.
CI runs pytest from the repo root but 'tests' is not an installable
package — it has no __init__.py at the top level — so absolute imports
like 'from tests.integration.conftest import ...' fail with
ModuleNotFoundError. Switch to relative imports ('from .conftest import
...') which work both in CI and locally regardless of PYTHONPATH.
…nit idempotency, atomic nft write)
1 participant
Summary
Replaces the flat worker model (
bare host → incus → agent containers) with a nested model: bare host runs one privileged LXC namedtaos-workerwith nested incus inside; agent containers live in there. Linux only in v1; macOS (Apple Containerization) and Windows (WSL2) are deferred to follow-up specs.This is the prerequisite for the agent-migration spec drafted at
docs/superpowers/specs/2026-05-05-agent-migration-design.md— that spec is paused until this lands.Why
Three concrete wins:
incus storage create taos-worker-pool btrfs size=Ncaps everything taOS can ever consume on the host. Host never runs out of disk because of taOS.What's in the PR
Backend (T1-T4, T9-T10):
WorkerInfo+WorkerRegister+HeartbeatBodygain capacity fields:host_lan_ip,storage_cap_bytes,storage_used_bytes,bytes_deduped_total,worker_lxc_image_version. Heartbeat reports the three byte counters.tinyagentos/cluster/worker_capacity.pyreads btrfs pool size + bees dedup totals (best-effort, returns zeros on any error).host_lan_ipcollision check (legacy flat-mode workers without the field skip the check). Privilege/nesting verification at incus-enroll flagsdegradedworkers.tinyagentos/cluster/convert_to_lxc.pyfor flat-to-LXC migration (drain, delete, redeploy).tinyagentos/cli/worker.pyexposestaos-worker-ctl convert-to-lxc | dedup enable\|disable | resize-storage.install-worker.sh two-phase rewrite (T5-T8):
TAOS_INSIDE_WORKERenv var.taos-agent-*containers; user runstaos-worker-ctl convert-to-lxcto migrate first.incus admin init --minimal→ btrfs storage pool (incus storage create taos-worker-pool btrfs size=$cap) → launch privileged+nestingtaos-workerLXC → nftables port-forward:8443→incus execre-exec into LXC.incus admin init --minimal→ register with controller includinghost_lan_ip+worker_lxc_image_version.Tests:
TAOS_INTEGRATION=1. Cover: fresh worker-LXC install, convert flat-to-LXC, disk-quota enforcement, dedup byte tracking.End-to-end verification
Verified against fresh Ubuntu 24.04 KVM VMs on the Fedora dev box throughout T5-T8 implementation. The 6 fix commits (
8069064,95c6113,4db70af,741f658,e506b0e,9ac924d) are real bugs the iterative testing surfaced —incus storage create source=failing on ext4 hosts, IP-discovery race against DHCP,incus list -c namecolumn expansion, stdin pollution from curl-pipe-bash, function-definition order at parse time, and bees not in Ubuntu Noble repos.End-to-end test passing on KVM (controller + worker host VMs):
security.privileged=trueandsecurity.nesting=truenft list rulesetshows:8443dnat to worker LXC IPincus exec taos-worker -- incus listhost_lan_ipandworker_lxc_image_versionfields populatedTest Plan
pytest tests/cluster/ tests/cli/→ 53/53 passingTAOS_INTEGRATIONnot set (4 skipped)bash -n scripts/install-worker.shcleantests/integration/withTAOS_INTEGRATION=1against the Fedora KVM box (manual; convert-to-LXC and disk-quota paths still need the real-cluster validation)Out of scope (deferred)
taos-worker-ctlinto a unifiedtaos worker ...dispatcher (currently a standalone console script)Spec
docs/superpowers/specs/2026-05-05-worker-as-lxc-design.md(gitignored; in your local repo).