Conversation
The download half of code sync — the missing piece for LocalKit's
"import this remote site as a new local site" flow (LocalKit plan 18).
- GET /pull/code streams a tar.gz of the site's wp-content, built inside
the container with `docker exec tar czf -` so permissions and structure
survive. Entries are prefixed `wp-content/`, the same format POST
/push/code accepts, so both directions share one archive shape. GNU
tar's exit 1 ("file changed as we read it") is tolerated — a live site
writing uploads mid-archive is normal.
- GET /pair now reports a `features` array so clients gate their UI on
capability instead of discovering a 404 mid-operation.
- GET /sites gains `php_version` (regexed off the compose image tag, not
a per-site container shell) and an explicit `site_url`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sync v1 POSTs the whole payload in one request, so it is capped by the panel's 100MB MAX_CONTENT_LENGTH and a drop at 99% means starting over. v2 makes every request one 8 MiB chunk, which lifts the cap and makes a retry cost only what was actually lost. - POST /push/<kind>/init: describes the transfer, reaps stale ones, and returns the offsets already confirmed for an identical payload — that array IS resume. - PUT /push/<kind>/chunk: per-chunk sha256, idempotent re-sends, marker written after the bytes so a crash costs one chunk, not a corrupt file. - POST /push/<kind>/finish: verifies the whole-file sha256, THEN runs the processing. Missing chunks are a 409 with the received set (resumable); a hash mismatch discards. Because processing only ever runs on verified bytes, an abandoned transfer cannot leave half-applied state. - push/code and push/db v1 now delegate to _install_code/_import_db, so v1 and v2 share exactly one processing path and the safe-extract policy stays mandatory in both. - Pulls accept ?session= and serve with conditional=True, giving Range / If-Range. The session pins one materialized export: pull/db and pull/code build their payload per request, and ranges taken from two different mysqldump/tar runs would assemble into neither. No session = v1 export-and-delete, unchanged. A transfer is a directory under the temp dir with per-chunk marker files rather than a shared JSON list, so concurrent chunk writes need no lock. Transfer ids are validated against uuid4().hex before touching a path. FEATURES gains 'sync-v2'; older clients keep the v1 routes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduces a full-stack Test Sandbox feature to run distro matrix tests in Docker from the UI: new backend API/service, sandbox_runs model + migration, run/log endpoints, and a new frontend page with mode selection, run history, live polling, cancel, and per-distro logs. Also adds ARM64 scripts CI coverage (ubuntu-24.04-arm) and strengthens install/test automation for minimal images and overlay rebuilds: early curl/bootstrap index refresh, stricter venv validation, release-venv glibc fallback, Node reprovisioning on source fallback, rsync --delete overlay behavior, npm bootstrap in VM install, Windows tar pinning in PowerShell harness, plus updated test docs and distro coverage notes.
DeploymentRunner.run() logged self.job.target_server_name, but the model only exposed that value as a to_dict() key, so every deploy job crashed with AttributeError at its first step and was marked failed before any work ran. Add a target_server_name property and reuse it in to_dict().
Flip DEFAULT_REGISTRY_URL to https://serverkit.ai/ext/index.json (the long-planned go-live default: cached proxy of the raw index with logo URL rewriting). Add logo fields to the bundled offline index so Marketplace cards get real art even before the first successful fetch, and update the registry tests + docs for the new default.
There was a problem hiding this comment.
Pull request overview
Adds an admin-facing “Test Sandbox” feature to run installer/script suites across a Docker distro matrix from the panel, while also hardening installer/test harness paths, making LocalKit sync resumable (chunked v2 + pull-code), and switching the Marketplace default registry to serverkit.ai.
Changes:
- Introduces Test Sandbox backend model/service/API plus a new React page (with SCSS + API client) to run and inspect per-distro Docker-based test runs.
- Implements LocalKit sync v2 (chunked/resumable uploads + resumable downloads) and adds pull-code + capability advertisement.
- Hardens install/test workflows (curl/bootstrap tooling, glibc/venv sanity check, Node provisioning) and updates registry default + logos + CI coverage.
Reviewed changes
Copilot reviewed 28 out of 28 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| VERSION | Bumps panel version. |
| scripts/test/vm-install.sh | Makes overlay mirroring stricter and bootstraps Node for release-overlay rebuilds. |
| scripts/test/README.md | Updates distro coverage docs and defaults vs opt-in targets. |
| scripts/test/full-stack-test.ps1 | Expands opt-in distro map and pins Windows tar behavior. |
| install.sh | Adds bootstrap tooling, strengthens venv/ensurepip checks, and improves source-fallback robustness. |
| frontend/src/styles/pages/_test-sandbox.scss | Adds Test Sandbox page styling (infra console look). |
| frontend/src/styles/main.scss | Imports the new Test Sandbox page stylesheet. |
| frontend/src/services/api/testSandbox.js | Adds frontend API helpers for Test Sandbox + raw log fetch. |
| frontend/src/services/api/index.js | Registers Test Sandbox API module into ApiService. |
| frontend/src/pages/TestSandbox.jsx | Adds the Test Sandbox admin page (runs, polling, logs, history). |
| frontend/src/components/sidebarItems.js | Adds “Test Sandbox” nav item under System. |
| frontend/src/App.jsx | Wires Test Sandbox route and page title. |
| docs/EXTENSIONS_REGISTRY.md | Updates registry docs to serverkit.ai default behavior. |
| builtin-extensions/serverkit-localkit/plugin.json | Bumps LocalKit extension version to 0.2.0. |
| builtin-extensions/serverkit-localkit/backend/localkit.py | Implements resumable sync v2 + pull-code + capabilities and site metadata enrichment. |
| backend/tests/test_registry.py | Updates tests for new default registry URL + logo normalization expectations. |
| backend/smoke_sandbox.py | Adds a manual smoke runner for the Test Sandbox service. |
| backend/migrations/versions/075_sandbox_runs.py | Adds migration for sandbox run history storage. |
| backend/app/services/test_sandbox_service.py | Implements Docker-backed distro matrix execution + logs + cancellation. |
| backend/app/services/registry_service.py | Switches DEFAULT_REGISTRY_URL to https://serverkit.ai/ext/index.json. |
| backend/app/models/sandbox_run.py | Adds SandboxRun model for persisted run history/results. |
| backend/app/models/deployment_job.py | Fixes deploy crash by adding/reusing target_server_name property. |
| backend/app/models/init.py | Exposes SandboxRun in models package import surface. |
| backend/app/data/registry_index.json | Adds bundled logo fields for Marketplace offline/default art. |
| backend/app/api/test_sandbox.py | Adds REST endpoints for distros/runs/logs (viewer/admin gated). |
| backend/app/init.py | Registers the Test Sandbox blueprint. |
| backend/_dbcheck.py | Adds a local SQLite integrity helper script. |
| .github/workflows/scripts-ci.yml | Adds native ARM64 job for script suite coverage. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+25
to
+27
| def list_runs(): | ||
| limit = min(int(request.args.get('limit', 20)), 100) | ||
| return jsonify({'runs': TestSandboxService.list_runs(limit=limit)}) |
Comment on lines
+190
to
+194
| with _active_lock: | ||
| names = list(_active_containers.get(run_id, [])) | ||
| for name in names: | ||
| subprocess.run(['docker', 'rm', '-f', name], | ||
| capture_output=True, timeout=30) |
Comment on lines
+278
to
+280
| finally: | ||
| subprocess.run(['docker', 'rm', '-f', container], | ||
| capture_output=True, timeout=30) |
Comment on lines
+285
to
+290
| cmd = [ | ||
| 'docker', 'run', '--rm', '--name', container, | ||
| '-v', f'{REPO_ROOT}:/src', | ||
| '-v', f'{work_dir}:/work', | ||
| image, 'sh', '/work/run-quick.sh', | ||
| ] |
Comment on lines
+320
to
+327
| from app.models import WordPressSite | ||
| for entry in payload.get('sites') or []: | ||
| site = WordPressSite.query.get(entry.get('id')) if entry.get('id') else None | ||
| if site is None or not site.application: | ||
| continue | ||
| entry['php_version'] = _php_version(site) | ||
| entry['site_url'] = entry.get('url') or _site_url(site) | ||
| return jsonify(payload), 200 |
jhd3197
added a commit
that referenced
this pull request
Aug 6, 2026
These five workflows triggered on both `push: [dev, main]` and `pull_request: [dev, main]`, so one change was validated up to three times: on the dev push, again on the dev->main PR, and a third time on the post-merge push to main. Narrow them to `push: [dev]` + `pull_request: [main]`: - `pull_request: [dev]` was dead config. Every PR this repo has had targets main (checked back through #77, dependabot's included), so it never produced a run. - `push: [main]` was redundant for release-smoke: release.yml's build-release job runs that same scripts/build-release.sh for real, on the same commit, moments later. Also drops test-system-utils' `unit-tests` job. It ran `pytest tests/test_utils_system.py` (44 tests) that Backend CI's bare `pytest` already collects — there is no pytest.ini, addopts or collect_ignore narrowing collection. The distro matrix and the raw-subprocess audit stay; those are the parts Backend CI genuinely cannot do. Coverage of main is unchanged: a pull_request run tests the merge result, and release.yml still gates itself on the full backend suite. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The best way to find out your installer breaks on a bare Debian image is to build a tool that runs it on one — and then watch it break. That's the through-line of this PR. A new Test Sandbox drives ServerKit's installer and script suites across a Linux distro matrix inside Docker, straight from the panel, and it immediately earned its keep: several of the
install.shfixes here are annotated "found via Test Sandbox full mode" — a too-new glibc in the release venv, apython -m venv --helpthat lies about ensurepip, npm going missing on the source-build fallback. Alongside it, LocalKit's sync grows a chunked, resumable v2 so a transfer that dies at 99% costs one 8 MiB chunk instead of the whole thing, plus the download half of code sync that its "import a remote site locally" flow was missing. Rounding it out: the Marketplace registry flips to its long-plannedserverkit.aidefault with real logo art, and a one-line-but-fatal deploy-job crash gets fixed.Highlights
install.shend-to-end in a systemd container and health-probes the result. Live per-distro status, run history, per-distro logs, and cancel./pairalso advertises a capability list so the desktop app disables buttons an older panel can't serve, instead of failing mid-operation.serverkit.airegistry and ships real extension logos, so cards show art even before the first successful fetch and even on air-gapped panels.AttributeErrorbefore any work ran.Technical changes
Test Sandbox (new full-stack feature)
SandboxRunmodel + migration075_sandbox_runs(idempotent — guards on live schema sinceMigrationServicerunscreate_all()before Alembic).resultsis a per-distro JSON map; full logs live on disk underinstance/sandbox-runs/<id>/<distro>.log.TestSandboxServiceruns a distro matrix in Docker with two modes:quick(ashrunner thatbash -n-lints every shipped script and runs the source-able unit suites, mirroring thescripts-cicross-distro job) andfull(a privilegedgeerlingguysystemd container running realinstall.sh+ a/api/v1/system/healthprobe). Per-distro work runs on its own thread; containers are namedsk-sandbox-<run_id>-<distro>so cancel/stale-sweep can find them; a module-level lock guards the active-container map. Full mode waits forsystemctl is-system-runningand real DNS+HTTP (probed withpython3, since curl isn't in those images) before installing, to dodge the post-boot networkd window that caches empty apt lists.test_sandbox.pyblueprint at/api/v1/test-sandbox:GET /distros,GET/POST /runs,GET /runs/<id>,POST /runs/<id>/cancel,GET /runs/<id>/logs/<distro>(plain text). Reads are@viewer_required, mutations@admin_required; a single-run lock returns 409 if one's already going.TestSandbox.jsxpage +_test-sandbox.scss, wired intoApp.jsxroutes/titles,sidebarItems.js(System category),main.scss, and atestSandbox.jsAPI module (the log fetch bypasses the JSON-parsingrequest()for raw text, same pattern asai.js).smoke_sandbox.py(manual one-distro smoke) and_dbcheck.py(sqlite integrity helper) added for local verification.LocalKit sync v2 (chunked + resumable) —
builtin-extensions/serverkit-localkit/backend/localkit.py, bumped to0.2.0POST /push/<kind>/initdescribes a transfer, reaps stale ones, and returns already-confirmed offsets (that array is the resume state).PUT /push/<kind>/chunkverifies a per-chunk sha256, is idempotent on re-send, and writes its marker after the bytes so a crash costs one chunk.POST /push/<kind>/finishverifies the whole-file sha256 before processing — missing chunks 409 with the received set, a hash mismatch discards, so processing only ever runs on verified bytes and an abandoned transfer can't half-apply.push/codeandpush/dbnow delegate to_install_code/_import_db, so v1 and v2 share one processing path and the mandatory safe-extract policy.?session=and serve withconditional=True(Range / If-Range). The session pins one materialized export so ranges never assemble across two differentmysqldump/tarruns; no session = the unchanged v1 export-and-delete.uuid4().hexbefore touching a path.FEATURESgainssync-v2.LocalKit pull-code + capabilities
GET /pull/codestreams atar.gzof the site'swp-content, built inside the container (docker exec ... tar czf -) to preserve permissions/structure, entries prefixedwp-content/to match whatpush/codeaccepts. GNU tar's exit 1 ("file changed as we read it") is tolerated for live sites.GET /pairreturns afeaturesarray (append-only capability contract) so clients gate UI on capability.GET /sitesgainsphp_version(regexed off the compose image tag — a file read, not a per-site container shell) and an explicitsite_url.Registry go-live —
registry_service.py,registry_index.json,test_registry.py,docs/EXTENSIONS_REGISTRY.mdDEFAULT_REGISTRY_URLflips from the raw-GitHub index tohttps://serverkit.ai/ext/index.json(proxies the same index with caching + logo URL rewriting); raw GitHub stays documented as theSERVERKIT_REGISTRY_URLmanual fallback.logofields for all extensions so Marketplace cards have art before the first fetch. Tests and docs updated for the new default.Deploy job crash fix —
deployment_job.pyDeploymentRunner.run()readself.job.target_server_name, but that value only existed as ato_dict()key, so every deploy crashed withAttributeErroron step one. Added atarget_server_nameproperty and reused it into_dict().Installer / test-harness hardening
install.sh: newensure_bootstrap_tools()(refresh package index + install curl up front) called frommain();py_venv_ok()now also importsensurepip(Debian'spython -m venv --helppasses without it and only fails at creation);build_virtualenv()discards a prebuilt release venv whose interpreter won't run (release built on Ubuntu 24.04 hard-requires GLIBC_2.38);sync_source()refreshes the apt index before installing git on empty-index images; aprovision_nodere-run on the source-build fallback since the release path skips Node..github/workflows/scripts-ci.yml: newcross-arch-arm64job onubuntu-24.04-armrunning the unit suites natively (Raspberry Pi / ARM homelab proxy — the VM harness can't virtualize ARM on Windows Hyper-V).scripts/test/full-stack-test.ps1:-Onlynow selects from the fullDistroMap(opt-inalma9/debian11/opensuse15in addition tofedora) in the order listed; tar pinned to Windows'System32\tar.exeso a Git-Bash PATH doesn't shadow it with GNU tar (which readsC:\...as a remote host).scripts/test/vm-install.sh: overlay nowrsync -a --delete(mirror exactly — a staleserverkit-tramoplugin from an older release base was surviving a plain overlay and breakingvite build), protecting.env/instance/ssl/venv/logs; bootstraps Node 22 when the release install skipped it.scripts/test/README.md: distro coverage table split into default vs opt-in, plus ARM-CI notes.