feat: 完整实现 Docker Compose 管理与原生 1Panel 商店 - #3
Conversation
Add a Docker Compose runtime alongside the existing Kubernetes one, behind a stable Controller seam. Backend-focused MVP covering stages 0-3 of issue #2. Domain & seam (stage 0): - Stable Application/DesiredApplication/Task/Revision model; JSON stays forward-compatible with the existing useApps()/useAppDetail() read path. - Controller interface is the only HTTP-facing seam; runtimeAdapter (compose + kubernetes) is internal. K8s Manager migrated to kubernetesRuntime adapter (deploymentToAppInfo -> deploymentToApplication; phase aggregated in backend). Compose runtime (stage 1): - Lightweight Docker Engine client over net/http (unix socket and tcp:// via DOCKER_HOST) for read/observe/logs — deliberately no docker SDK dependency. - Writes go through `docker compose` CLI via exec.CommandContext with arg arrays (no shell). Only devbox-managed projects (prefix devbox-<id>) are - discovered; Docker downgrades cleanly without affecting K8s. Persistence & async tasks (stage 2): - SQLite (modernc.org/sqlite, pure Go) for app meta / tasks / revisions / idempotency / audit. Compose file stays the source of truth on disk. - Persistent Task worker: per-app serial queue, idempotency (same key+request -> same task; same key+different request -> 409), crash recovery on restart. - start/stop/restart/redeploy/remove; data kept by default, purge is explicit, external volumes are never removed. Create/edit (stage 3): - Inline Compose create/update; `compose config` preflight; risk policy (blocked/confirmation/warning); revision history + optimistic concurrency (expectedRevision mismatch -> 409). app ID path-traversal safe; Compose args shell-injection safe; secrets never returned or stored in task/revision/audit. HTTP & wiring: - Legacy read/action paths preserved as a compatibility shim (sync wait) so the existing UI is unchanged; new write APIs return 202+Task. New endpoints: validate, capability, tasks/{id}, actions/{...}, compose, revisions, revisions/{n}/restore, operations. - config: compose section (data_dir / docker_socket / enabled); main assembles the Controller (K8s + Compose adapters + sqlite + worker). UI (minimal): - New "Compose 应用" system entry -> ComposeManager page: runtime filter, lifecycle buttons, inline-Compose create dialog with preflight, task feedback. Desktop/AppMgmtDrawer intentionally untouched. Verified end-to-end against a real Docker daemon (remote TCP) via HTTP smoke: capability, validate, create(202), task poll->succeeded, list/detail (running, 2 services), stop/start (phase transitions), revision conflict 409, idempotency (202/202/409), delete->404. Real-docker integration test guarded by build tag `integration`; unit tests cover domain/risk/persistence/worker/HTTP. Out of scope (later stages): app-store unified install (stage 4), backup and multi-host (stage 5). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… reliability) Address two-axis self-review findings on the Issue #2 compose MVP. All HIGH and security/reliability MEDIUM items fixed; cheap-and-correct LOWs applied. Secrets never enter task/revision/audit/log/error. Standards/spec HIGH: - domain: Error carries Findings; RiskBlockedErr stores them (was discarded). HTTP returns secret-free findings JSON. - controller RestoreRevision + worker.execute: meta read errors/no-such-app are strict (no empty-ID Upsert polluting the table, no silent K8s fallback). - Apply/Validate run a REAL `docker compose config` precheck in an isolated temp dir (0700/0600, fixed temp project, 30s timeout, 1MB cap, no shell, cleaned up) BEFORE any file/revision/task is written. Apply hard-requires it (capability error if the compose CLI is missing; validation error if invalid). Risk analysis runs on the RENDERED content so ${VAR} cannot bypass privileged/socket/root bind/host-mode. Render stdout (may hold secret values) is kept in-memory only and discarded; errors expose only sanitized stderr. MEDIUM / LOW: - worker: recover panics per task (-> failed, keep consuming); Enqueue blocks on a full per-app queue respecting ctx (no dropped persisted tasks); re-Observe after apply/start/restart/redeploy and fail when containers are absent/failed; reclaim a removed app's queue+goroutine (stopQueue). - repository: CommitApply/CommitTask run revision+meta+task(+idem) in one tx; Apply uses staging + atomic rename promoted only after the DB commit, with compensation on the (near-impossible) rename failure. - docker_engine: ping honors ctx. - risk: relative `..` traversal bind -> confirmation; rendered analysis covers ${SOCK}:/var/run/docker.sock and ${ROOT}:/host. - kubernetes: ready==0 no longer auto-failed; deploymentPhase uses Available/Progressing(ProgressDeadlineExceeded)/ReplicaFailure; K8s Logs redacted via sanitizeLog. - controller/handler: uniform JSON error envelope; awaitTask watches request ctx; compose pull errors logged (no silent swallow); paths SafeWriteFile escape condition parenthesized; aggregatePhase emits failed for dead containers; repo clock injectable. Tests: new compose_cli/kubernetes/paths tests + worker/controller/repo/risk/ domain/handler regression tests. make test (-race) + make build + frontend build green; real Docker E2E (TestComposeE2E) green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
UI 验证已完成:本地浏览器实测「Compose 应用」页面显示 Docker Compose 29.5.3、运行时筛选与新建入口,未捕获页面错误。截图已生成于本地 |
|
部署验证更新(192.168.1.2):首次替换后发现 |
Backend (pkg/apps, pkg/console): - StoreManager.GetStoreAppVersion re-fetches the trusted catalog version and back-compat reads spec.composeTemplate/spec.compose; 4MiB response cap, URL escaping, sanitized/truncated errors. - Safe template rendering (store_render.go): text/template missingkey=error with NO FuncMap (no arbitrary funcs/shell/file); secrets stay in .env only and never enter compose/revision/audit; values validated against schema (required/type/ select), unknown keys rejected; 1MiB output cap. - HTTP /store/version returns valuesSchema/defaultValues/runtime/installable (compose template never serialized) and /store/install re-fetches -> validates -> renders -> Controller.Apply -> 202+Task; list gains runtime/installable. - Same catalog app is reused via ID + ExpectedRevision; version switch bumps revision. Frontend (AppStore.jsx): - Cards/detail show Compose/Kubernetes runtime; K8s-only apps show "Kubernetes only" and disable deploy (no ghost requests). - install polls via taskId/useTask (queued/running/succeeded/failed). - Clear empty states: loading / unconfigured / no match. Real edge-apiserver ApplicationVersionSpec has no compose field today, so live packages surface as Kubernetes runtime (CEO ruling #2); composeTemplate is a back-compat field ready to activate once edge-apiserver provides it. Fake catalog + stub-controller tests cover the full pipe. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Independent audit of 09e62a4 returned No BLOCKER but surfaced real correctness/safety issues; all fixed with tests: - H1 compareVersionStrings used lexicographic order, so 10.0.0 < 9.0.0 and maxVersionItem picked the wrong "latest" approved version. Switch to numeric compare per segment with string fallback (mirrors frontend compareVersions); add multi-digit tests. - M1 default install idempotency key excluded secrets (hashApplyRequest omits Secrets), so re-installing the same app+version with a rotated password short-circuited to the old task and the new secret never persisted, while changing params hit idempotency_conflict. New StoreInstallFingerprint folds params+secrets into the default key so only a truly identical request is idempotent. - L1/L3 parseValuesSchema now rejects field keys not matching ^[A-Za-z_][A-Za-z0-9_]*$ (env/template safety) and select fields without options. - L2 /store/install wraps the body in http.MaxBytesReader (1MiB). - M2 documented the list(provisioner) vs detail(composeTemplate) runtime mismatch; install re-checks via GetStoreAppVersion and 422s, so no wrong install, only a one-shot UX inconsistency. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
部署验证更新(192.168.1.2,Compose 商店版):已将 PR head 验证结果:
部署前 |
Add third-party Docker Compose catalog sources (HTTP + Git shallow clone) and complete the management-detail data model for Issue #2. Catalog sources (pkg/apps/catalog.go): - httpCatalog: HTTPS manifest + relative compose fetch (localhost/`.test`/ `.internal` or explicit insecure for http); bounded reads/responses/errors. - gitCatalog: controlled Git HTTP(S) shallow clone via exec.CommandContext (argv, no shell, --depth 1 --single-branch --no-tags); fixed timeout/output/ total-size; rejects file/ssh/git/local; token injected via http.extraHeader and scrubbed from all output; relative compose read via safeReadCatalogFile (Clean + EvalSymlinks + root prefix) to block traversal/symlink escape. - CatalogSet: multi-source aggregation with per-source failure isolation; serves last-good cache when a source is unreachable; does not affect installed apps. Atomic refresh for git (clone to temp, rename on success). Unified install (pkg/console/handlers_store.go, handlers_catalog.go): - Backend re-fetches version by sourceId+appId+version from the trusted source; frontend cannot supply compose (StoreAppVersion.ComposeTemplate is json:"-"). Shared installResolvedVersion path for store and catalog. Risk policy (pkg/apps/controller.go): - store/catalog packages with latest/main/master/edge/nightly image tags are blocked (project red line); blocked risks remain non-overridable; inline keeps explicit, audited confirmation override. Management detail (pkg/apps/inventory.go, controller.go, handlers_apps.go): - GET /apps/{id}/storage: volume inventory (managed/external/bind/socket) + managed data dir; external never deleted. - GET /apps/{id}/env: env metadata (key/configured/type/required, no values). - GET /apps/{id}/remove-preview?purge=: explicit willDelete/willKeep; default keeps data, external never deleted. Config + wiring (pkg/config/config.go, cmd/devbox/main.go): - compose.catalogs[] (http/git) + compose.catalog_poll; CatalogSet assembled and started in main; only read-only status/explicit refresh, no arbitrary URL write-in. Routes under /api/v1/catalogs*; all write paths go through Controller.Apply (read-only catalog proxies do not mutate app state). Tests: httptest HTTP catalog, git temp-repo clone integration, cache/offline, size limits, URL scheme/credential/path/symlink, source conflict, trusted refetch, risk rejection, secret non-leak, storage inventory/external purge. gofmt clean, go vet clean, go test -race passes. Issue #2 body updated to current scope (third-party catalog delivered; multi-host / reliable backup / GitOps auto-sync / K8s<->Compose conversion recorded as Issue-self-triggered phase-5 conditions). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
最终部署验证(192.168.1.2,PR head
首次 health 请求发生在 systemd 刚进入 active、9092 尚未监听的瞬间;等待启动后一次合并检查全部通过,未触发回滚。 |
|
最终部署与真实 Docker 验证(192.168.1.2,PR head
UI 截图仍未擅自上传:登录测试环境和 GitHub user-attachments 上传需要用户明确授权。 |
|
原生 1Panel 商店与最终部署验证(PR head
本地执行机对 GitHub 443 的一次官方源测试因网络超时失败;相同测试二进制在目标机对官方源通过。PR 保持 open,等待人工 review,不会自动 merge。 |
|
自动发现与安全接管版本已更新并部署(PR head
Codex/GPT-5.6-sol xhigh 预 review 因当前 Codex token 401 且 read-only bubblewrap 无法创建 namespace 未完成;限制已写入 PR body。独立 Standards/Spec 自审均无 BLOCKER/HIGH。PR 保持 open,等待人工 review,不会自动 merge。 |
关联 Issue
Closes #2
结果
完整交付单机 Docker Compose 应用管理、原生 1Panel 开源应用商店兼容,以及系统现有 Compose project 的自动发现与安全接管。PR 保持 open,等待人工 review;不会自动 merge。
本次更新:系统 Compose 自动发现与接管
all=true扫描所有仍保留容器记录(含 stopped)的 Compose project,按 project/service 聚合 phase、health、replica、ports 与 endpoints。not_managed。POST /api/v1/apps/{id}/takeover重新从 daemon 获取 source labels,不信任客户端路径;保留原 Compose project name,接管后复用现有编辑、revision、Task、日志、生命周期和卸载边界。OriginalProject存入 SQLite,进程重启后仍操作原 project,named volume 不改名。compose down且容器记录删除后,daemon 无 metadata 可供发现;UI/文档已说明,可改用粘贴/上传导入。接管安全边界
openat2:working dir 禁 symlink/magic-link;config files 必须位于其下、禁 symlink/跨文件系统/非普通文件/超限/读取中变化;FIFO 以 non-blocking 方式打开并立即拒绝。非 Linux 明确返回 capability 错误。include、extends.file、env_file、secrets.file、configs.file、build。-H,不继承 HOME、DOCKER_HOST、proxy、token 等控制面环境。既有交付
apps.Controller/ Application / Task / Revision 领域 seam;Compose/Kubernetes runtime adapter 隐藏。devbox/v1HTTP/Git catalog 与原生 1Panel 官方目录统一走可信安装链路。双轴自审
Standards
无 BLOCKER/HIGH。重点核对了未接管写保护、openat2 路径安全、CLI 环境隔离、secret 落盘/回显、SQLite 原子性、per-app 锁和原 project 持久化。审查中发现 FIFO config path 可能在判定“非普通文件”前阻塞,已增加
O_NONBLOCK和 Linux 回归测试。Spec
无 BLOCKER/HIGH。自动发现 stopped project、统一展示、只读诊断、显式接管、保留原 project、接管后可编辑、重启恢复、volume 数据保持和完全 down 边界均有实现与测试证据。
指定 Codex runtime +
gpt-5.6-sol/xhigh预 review 未完成:当前 Codex refresh/access token 失效(401),且 read-only bubblewrap 无法创建 namespace;已按 workflow 记录限制,没有伪造 review 结论或引入其它模型替代。验证
git diff --cached --check:通过go test ./pkg/apps ./pkg/console -race -count=1:通过make test:通过(Go race;pkg/apps71.6%)go vet ./...:通过npm test:7/7npm run build:通过(仅既有 >500 kB chunk 提示)make build-all:通过,Linux amd64 二进制与嵌入 UI 已同步go test -tags=integration ./pkg/apps -run '^$':通过TestTakeoverE2EPASS(48.89s),覆盖发现→接管→编辑/Apply→进程重启→restart、原 project name 与 named volume marker 保持,以及独立 project 完全 down 后不再发现;测试资源均已清理。UI 有用户可见变更;测试环境启用了认证,且当前没有已授权的 GitHub user-attachments 浏览器上传会话,因此未擅自登录或上传截图。没有使用 gist、Release 或提交截图文件替代。
明确边界