Skip to content

feat: 完整实现 Docker Compose 管理与原生 1Panel 商店 - #3

Open
wuxs wants to merge 14 commits into
mainfrom
issue-2-compose-apps
Open

feat: 完整实现 Docker Compose 管理与原生 1Panel 商店#3
wuxs wants to merge 14 commits into
mainfrom
issue-2-compose-apps

Conversation

@wuxs

@wuxs wuxs commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

关联 Issue

Closes #2

结果

完整交付单机 Docker Compose 应用管理、原生 1Panel 开源应用商店兼容,以及系统现有 Compose project 的自动发现与安全接管。PR 保持 open,等待人工 review;不会自动 merge。

本次更新:系统 Compose 自动发现与接管

  • Docker Engine all=true 扫描所有仍保留容器记录(含 stopped)的 Compose project,按 project/service 聚合 phase、health、replica、ports 与 endpoints。
  • 与 SQLite 受管应用、Kubernetes observed ID 去重;discovered ID 按 project 排序确定性分配,List/Get/Takeover 一致。
  • 外部 project 标记“已发现 · 只读”,仅开放日志和显式“接管并编辑”;未接管 start/stop/restart/redeploy/remove/restore 全部返回 not_managed
  • POST /api/v1/apps/{id}/takeover 重新从 daemon 获取 source labels,不信任客户端路径;保留原 Compose project name,接管后复用现有编辑、revision、Task、日志、生命周期和卸载边界。
  • 接管源目录只读;多文件 Compose 归一化成 devbox 托管副本,不修改原文件。OriginalProject 存入 SQLite,进程重启后仍操作原 project,named volume 不改名。
  • UI 展示 project/service/state/health/ports、来源路径和不可接管原因;接管弹层支持 Esc、focus trap、focus return、busy guard 与风险确认。
  • 已知边界:外部 project 完全 compose down 且容器记录删除后,daemon 无 metadata 可供发现;UI/文档已说明,可改用粘贴/上传导入。

接管安全边界

  • Linux 使用 openat2:working dir 禁 symlink/magic-link;config files 必须位于其下、禁 symlink/跨文件系统/非普通文件/超限/读取中变化;FIFO 以 non-blocking 方式打开并立即拒绝。非 Linux 明确返回 capability 错误。
  • Compose CLI 前预扫描并阻断 includeextends.fileenv_filesecrets.fileconfigs.filebuild
  • 原始 Compose 中明文 secret 在任何临时落盘前阻断;临时目录 0700、文件 0600、显式空 env-file。
  • Docker CLI 子进程使用固定 PATH/locale,daemon 端点走 -H,不继承 HOME、DOCKER_HOST、proxy、token 等控制面环境。
  • blocked 风险不可 override;confirmation 需显式确认。接管 task、幂等指纹和 audit 不记录宿主 config path 或 secret。
  • staging 写 compose/revision/marker 并 fsync,原子 rename 后再提交 SQLite;崩溃 orphan 仅在 marker 与 project/hash 一致时恢复。

既有交付

  • apps.Controller / Application / Task / Revision 领域 seam;Compose/Kubernetes runtime adapter 隐藏。
  • Compose CLI v2 写 + Docker Engine API 读;SQLite 持久任务、revision、幂等、audit;异步生命周期、健康等待、乐观并发和崩溃恢复。
  • 粘贴/上传、预检、结构化参数、Compose 编辑、定义回退、8-tab 管理详情;默认 remove 保留数据,purge 不删 external/bind/socket。
  • edge-apiserver storeapps、通用 devbox/v1 HTTP/Git catalog 与原生 1Panel 官方目录统一走可信安装链路。

双轴自审

Standards

无 BLOCKER/HIGH。重点核对了未接管写保护、openat2 路径安全、CLI 环境隔离、secret 落盘/回显、SQLite 原子性、per-app 锁和原 project 持久化。审查中发现 FIFO config path 可能在判定“非普通文件”前阻塞,已增加 O_NONBLOCK 和 Linux 回归测试。

Spec

无 BLOCKER/HIGH。自动发现 stopped project、统一展示、只读诊断、显式接管、保留原 project、接管后可编辑、重启恢复、volume 数据保持和完全 down 边界均有实现与测试证据。

指定 Codex runtime + gpt-5.6-sol/xhigh 预 review 未完成:当前 Codex refresh/access token 失效(401),且 read-only bubblewrap 无法创建 namespace;已按 workflow 记录限制,没有伪造 review 结论或引入其它模型替代。

验证

  • git diff --cached --check:通过
  • go test ./pkg/apps ./pkg/console -race -count=1:通过
  • make test:通过(Go race;pkg/apps 71.6%)
  • go vet ./...:通过
  • npm test:7/7
  • 目标前端 ESLint:0 error
  • npm run build:通过(仅既有 >500 kB chunk 提示)
  • make build-all:通过,Linux amd64 二进制与嵌入 UI 已同步
  • go test -tags=integration ./pkg/apps -run '^$':通过
  • 真实 Docker:TestTakeoverE2E PASS(48.89s),覆盖发现→接管→编辑/Apply→进程重启→restart、原 project name 与 named volume marker 保持,以及独立 project 完全 down 后不再发现;测试资源均已清理。

UI 有用户可见变更;测试环境启用了认证,且当前没有已授权的 GitHub user-attachments 浏览器上传会话,因此未擅自登录或上传截图。没有使用 gist、Release 或提交截图文件替代。

明确边界

  • CasaOS 官方 v2 协议已评估,但构建、路径和参数语义不同,留后续独立 adapter。
  • fnOS / Unraid 本轮不在范围,不作未经核验的开闭源断言。
  • 多主机调度、可靠数据备份/恢复、GitOps 自动 reconcile、K8s↔Compose 自动转换不属于当前单机里程碑。

wuxs and others added 5 commits July 22, 2026 02:25
Add a Docker Compose runtime alongside the existing Kubernetes one, behind a
stable Controller seam. Backend-focused MVP covering stages 0-3 of issue #2.

Domain & seam (stage 0):
- Stable Application/DesiredApplication/Task/Revision model; JSON stays
  forward-compatible with the existing useApps()/useAppDetail() read path.
- Controller interface is the only HTTP-facing seam; runtimeAdapter (compose +
  kubernetes) is internal. K8s Manager migrated to kubernetesRuntime adapter
  (deploymentToAppInfo -> deploymentToApplication; phase aggregated in backend).

Compose runtime (stage 1):
- Lightweight Docker Engine client over net/http (unix socket and tcp:// via
  DOCKER_HOST) for read/observe/logs — deliberately no docker SDK dependency.
- Writes go through `docker compose` CLI via exec.CommandContext with arg
  arrays (no shell). Only devbox-managed projects (prefix devbox-<id>) are
- discovered; Docker downgrades cleanly without affecting K8s.

Persistence & async tasks (stage 2):
- SQLite (modernc.org/sqlite, pure Go) for app meta / tasks / revisions /
  idempotency / audit. Compose file stays the source of truth on disk.
- Persistent Task worker: per-app serial queue, idempotency (same key+request
  -> same task; same key+different request -> 409), crash recovery on restart.
- start/stop/restart/redeploy/remove; data kept by default, purge is explicit,
  external volumes are never removed.

Create/edit (stage 3):
- Inline Compose create/update; `compose config` preflight; risk policy
  (blocked/confirmation/warning); revision history + optimistic concurrency
  (expectedRevision mismatch -> 409). app ID path-traversal safe; Compose args
  shell-injection safe; secrets never returned or stored in task/revision/audit.

HTTP & wiring:
- Legacy read/action paths preserved as a compatibility shim (sync wait) so the
  existing UI is unchanged; new write APIs return 202+Task. New endpoints:
  validate, capability, tasks/{id}, actions/{...}, compose, revisions,
  revisions/{n}/restore, operations.
- config: compose section (data_dir / docker_socket / enabled); main assembles
  the Controller (K8s + Compose adapters + sqlite + worker).

UI (minimal):
- New "Compose 应用" system entry -> ComposeManager page: runtime filter,
  lifecycle buttons, inline-Compose create dialog with preflight, task
  feedback. Desktop/AppMgmtDrawer intentionally untouched.

Verified end-to-end against a real Docker daemon (remote TCP) via HTTP smoke:
capability, validate, create(202), task poll->succeeded, list/detail (running,
2 services), stop/start (phase transitions), revision conflict 409, idempotency
(202/202/409), delete->404. Real-docker integration test guarded by build tag
`integration`; unit tests cover domain/risk/persistence/worker/HTTP.

Out of scope (later stages): app-store unified install (stage 4), backup and
multi-host (stage 5).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… reliability)

Address two-axis self-review findings on the Issue #2 compose MVP. All HIGH
and security/reliability MEDIUM items fixed; cheap-and-correct LOWs applied.
Secrets never enter task/revision/audit/log/error.

Standards/spec HIGH:
- domain: Error carries Findings; RiskBlockedErr stores them (was discarded).
  HTTP returns secret-free findings JSON.
- controller RestoreRevision + worker.execute: meta read errors/no-such-app are
  strict (no empty-ID Upsert polluting the table, no silent K8s fallback).
- Apply/Validate run a REAL `docker compose config` precheck in an isolated
  temp dir (0700/0600, fixed temp project, 30s timeout, 1MB cap, no shell,
  cleaned up) BEFORE any file/revision/task is written. Apply hard-requires it
  (capability error if the compose CLI is missing; validation error if invalid).
  Risk analysis runs on the RENDERED content so ${VAR} cannot bypass
  privileged/socket/root bind/host-mode. Render stdout (may hold secret values)
  is kept in-memory only and discarded; errors expose only sanitized stderr.

MEDIUM / LOW:
- worker: recover panics per task (-> failed, keep consuming); Enqueue blocks on
  a full per-app queue respecting ctx (no dropped persisted tasks); re-Observe
  after apply/start/restart/redeploy and fail when containers are absent/failed;
  reclaim a removed app's queue+goroutine (stopQueue).
- repository: CommitApply/CommitTask run revision+meta+task(+idem) in one tx;
  Apply uses staging + atomic rename promoted only after the DB commit, with
  compensation on the (near-impossible) rename failure.
- docker_engine: ping honors ctx.
- risk: relative `..` traversal bind -> confirmation; rendered analysis covers
  ${SOCK}:/var/run/docker.sock and ${ROOT}:/host.
- kubernetes: ready==0 no longer auto-failed; deploymentPhase uses
  Available/Progressing(ProgressDeadlineExceeded)/ReplicaFailure; K8s Logs
  redacted via sanitizeLog.
- controller/handler: uniform JSON error envelope; awaitTask watches request
  ctx; compose pull errors logged (no silent swallow); paths SafeWriteFile
  escape condition parenthesized; aggregatePhase emits failed for dead
  containers; repo clock injectable.

Tests: new compose_cli/kubernetes/paths tests + worker/controller/repo/risk/
domain/handler regression tests. make test (-race) + make build + frontend
build green; real Docker E2E (TestComposeE2E) green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wuxs

wuxs commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator Author

UI 验证已完成:本地浏览器实测「Compose 应用」页面显示 Docker Compose 29.5.3、运行时筛选与新建入口,未捕获页面错误。截图已生成于本地 /tmp/devbox-issue2-compose-manager.png;当前浏览器自动化 skill 对文件上传要求额外显式确认,因此本轮未擅自上传附件,也未把截图提交到分支或使用 Release/gist 图床。

@wuxs

wuxs commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator Author

部署验证更新(192.168.1.2):首次替换后发现 /var/lib/devbox 不存在导致 SQLite Controller 降级,已补 AssembleController 自动创建数据目录及回归测试(commit 1afa72a)。重新编译并原子替换 /opt/devbox/devbox 后验证:devbox.service=active、启动版本 1afa72a/var/lib/devbox/apps.db 已创建、日志显示 Compose runtime enabled、公开 /api/v1/health 返回 ok。远端启用了认证,未在评论或命令输出中暴露凭证;K8s runtime 因当前机器无集群配置按设计降级。远端保留备份 /opt/devbox/devbox.backup-20260722-154743/opt/devbox/devbox.backup-20260722-153638

wuxs and others added 2 commits July 22, 2026 16:40
Backend (pkg/apps, pkg/console):
- StoreManager.GetStoreAppVersion re-fetches the trusted catalog version and
  back-compat reads spec.composeTemplate/spec.compose; 4MiB response cap, URL
  escaping, sanitized/truncated errors.
- Safe template rendering (store_render.go): text/template missingkey=error with
  NO FuncMap (no arbitrary funcs/shell/file); secrets stay in .env only and never
  enter compose/revision/audit; values validated against schema (required/type/
  select), unknown keys rejected; 1MiB output cap.
- HTTP /store/version returns valuesSchema/defaultValues/runtime/installable
  (compose template never serialized) and /store/install re-fetches -> validates
  -> renders -> Controller.Apply -> 202+Task; list gains runtime/installable.
- Same catalog app is reused via ID + ExpectedRevision; version switch bumps revision.

Frontend (AppStore.jsx):
- Cards/detail show Compose/Kubernetes runtime; K8s-only apps show
  "Kubernetes only" and disable deploy (no ghost requests).
- install polls via taskId/useTask (queued/running/succeeded/failed).
- Clear empty states: loading / unconfigured / no match.

Real edge-apiserver ApplicationVersionSpec has no compose field today, so live
packages surface as Kubernetes runtime (CEO ruling #2); composeTemplate is a
back-compat field ready to activate once edge-apiserver provides it. Fake catalog
+ stub-controller tests cover the full pipe.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Independent audit of 09e62a4 returned No BLOCKER but surfaced real
correctness/safety issues; all fixed with tests:

- H1 compareVersionStrings used lexicographic order, so 10.0.0 < 9.0.0
  and maxVersionItem picked the wrong "latest" approved version. Switch to
  numeric compare per segment with string fallback (mirrors frontend
  compareVersions); add multi-digit tests.
- M1 default install idempotency key excluded secrets (hashApplyRequest
  omits Secrets), so re-installing the same app+version with a rotated
  password short-circuited to the old task and the new secret never
  persisted, while changing params hit idempotency_conflict. New
  StoreInstallFingerprint folds params+secrets into the default key so only
  a truly identical request is idempotent.
- L1/L3 parseValuesSchema now rejects field keys not matching
  ^[A-Za-z_][A-Za-z0-9_]*$ (env/template safety) and select fields without
  options.
- L2 /store/install wraps the body in http.MaxBytesReader (1MiB).
- M2 documented the list(provisioner) vs detail(composeTemplate) runtime
  mismatch; install re-checks via GetStoreAppVersion and 422s, so no wrong
  install, only a one-shot UX inconsistency.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wuxs

wuxs commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator Author

部署验证更新(192.168.1.2,Compose 商店版):已将 PR head c082998 编译为 Linux amd64 二进制并原子替换 /opt/devbox/devbox

验证结果:

  • devbox.service=active,MainPID 2037197
  • 启动日志明确显示 version=c082998Compose runtime enabled
  • GET /api/v1/health 返回 HTTP 200,状态为 ok
  • 未认证访问 /api/v1/store/apps 返回预期 401(该环境已启用本地密码认证)。
  • 远端二进制 SHA256 为 13766f152ee34a4f01bb4b80f7eeca31727993a50db3afbd9f64486e0524465e,与本地构建一致。
  • 已保留可恢复备份 /opt/devbox/devbox.backup-20260722-180019

部署前 make test(race)和 make build-all 均通过。

wuxs and others added 3 commits July 22, 2026 23:57
Add third-party Docker Compose catalog sources (HTTP + Git shallow clone)
and complete the management-detail data model for Issue #2.

Catalog sources (pkg/apps/catalog.go):
- httpCatalog: HTTPS manifest + relative compose fetch (localhost/`.test`/
  `.internal` or explicit insecure for http); bounded reads/responses/errors.
- gitCatalog: controlled Git HTTP(S) shallow clone via exec.CommandContext
  (argv, no shell, --depth 1 --single-branch --no-tags); fixed timeout/output/
  total-size; rejects file/ssh/git/local; token injected via http.extraHeader
  and scrubbed from all output; relative compose read via safeReadCatalogFile
  (Clean + EvalSymlinks + root prefix) to block traversal/symlink escape.
- CatalogSet: multi-source aggregation with per-source failure isolation;
  serves last-good cache when a source is unreachable; does not affect
  installed apps. Atomic refresh for git (clone to temp, rename on success).

Unified install (pkg/console/handlers_store.go, handlers_catalog.go):
- Backend re-fetches version by sourceId+appId+version from the trusted
  source; frontend cannot supply compose (StoreAppVersion.ComposeTemplate is
  json:"-"). Shared installResolvedVersion path for store and catalog.

Risk policy (pkg/apps/controller.go):
- store/catalog packages with latest/main/master/edge/nightly image tags are
  blocked (project red line); blocked risks remain non-overridable; inline
  keeps explicit, audited confirmation override.

Management detail (pkg/apps/inventory.go, controller.go, handlers_apps.go):
- GET /apps/{id}/storage: volume inventory (managed/external/bind/socket)
  + managed data dir; external never deleted.
- GET /apps/{id}/env: env metadata (key/configured/type/required, no values).
- GET /apps/{id}/remove-preview?purge=: explicit willDelete/willKeep; default
  keeps data, external never deleted.

Config + wiring (pkg/config/config.go, cmd/devbox/main.go):
- compose.catalogs[] (http/git) + compose.catalog_poll; CatalogSet assembled
  and started in main; only read-only status/explicit refresh, no arbitrary
  URL write-in.

Routes under /api/v1/catalogs*; all write paths go through Controller.Apply
(read-only catalog proxies do not mutate app state).

Tests: httptest HTTP catalog, git temp-repo clone integration, cache/offline,
size limits, URL scheme/credential/path/symlink, source conflict, trusted
refetch, risk rejection, secret non-leak, storage inventory/external purge.
gofmt clean, go vet clean, go test -race passes.

Issue #2 body updated to current scope (third-party catalog delivered;
multi-host / reliable backup / GitOps auto-sync / K8s<->Compose conversion
recorded as Issue-self-triggered phase-5 conditions).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wuxs wuxs changed the title feat: 实现 Docker Compose 应用管理 MVP feat: 完整实现 Docker Compose 应用管理与第三方商店 Jul 22, 2026
@wuxs

wuxs commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator Author

最终部署验证(192.168.1.2,PR head ca573e1):

  • 已从干净 commit 运行 make build-all,Linux amd64 二进制版本为 ca573e1
  • 已单次流式上传、校验 SHA256 后原子替换 /opt/devbox/devbox 并重启 devbox.service
  • devbox.service=activeGET /api/v1/health 返回 200。
  • 未认证访问 /api/v1/catalogs 返回预期 401(该环境启用了本地认证)。
  • 日志显示 Compose runtime enabled,data dir /var/lib/devbox,Docker socket /var/run/docker.sock
  • 远端 SHA256 50cf76b5593be6d797392f8dd1bf149fbe7c9014698835ea4595248d6a07e1cf,与本地构建一致。
  • 浏览器登录页显示 Edge Platform ca573e1,证明新二进制及嵌入 UI 正在提供服务。
  • 保留可恢复备份 /opt/devbox/devbox.backup-20260722-1928

首次 health 请求发生在 systemd 刚进入 active、9092 尚未监听的瞬间;等待启动后一次合并检查全部通过,未触发回滚。

@wuxs

wuxs commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator Author

最终部署与真实 Docker 验证(192.168.1.2,PR head 046601b):

  • 最终 commit 后重新编译 integration test binary,并在目标机本地 Docker daemon 上运行:TestComposeE2E PASS(24.56s),TestComposeVolumeLifecycleAndSettingsE2E PASS(12.13s)。后者真实验证结构化 CPU/内存/restart 设置、managed volume purge、external volume 永不删除。
  • 只有 E2E 全部通过后才原子替换 /opt/devbox/devbox 并重启 devbox.service
  • devbox.service=activeGET /api/v1/health 返回 200;未认证 GET /api/v1/catalogs 返回预期 401。
  • 启动日志显示 Compose runtime enabled,Docker socket /var/run/docker.sock
  • 本地与远端二进制 SHA256 均为 8e1aa14fc7d0078d1d8706f63f6f317f2fc996eec52bda66813ce5f52eb01d46
  • 可恢复备份:/opt/devbox/devbox.backup-20260723-issue2
  • systemd 刚重启时第一次 health 探测短暂连接失败,就绪循环随后成功;未触发回滚。

UI 截图仍未擅自上传:登录测试环境和 GitHub user-attachments 上传需要用户明确授权。

@wuxs wuxs changed the title feat: 完整实现 Docker Compose 应用管理与第三方商店 feat: 完整实现 Docker Compose 管理与原生 1Panel 商店 Jul 23, 2026
@wuxs

wuxs commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

原生 1Panel 商店与最终部署验证(PR head b71c2c7):

  • 目标机直接读取官方 https://github.com/1Panel-dev/appstoreTestOnePanelOfficialCatalogIntegration PASS(3.51s),覆盖官方默认分支、应用数量、adminer 5.5.0-standalone、参数 schema、container_name 剥离和 external network 收敛。
  • make test(race,pkg/apps 71.2%)、go vet ./...、目标 ESLint、前端 helper tests 5/5、Vite build、make build-all 全部通过。
  • 已原子替换 192.168.1.2:/opt/devbox/devboxdevbox.service=active,health 200,日志显示 Compose runtime enabledCatalog sources started
  • 本地/远端 SHA256 均为 ce19708223f37d916f2e7abb4a6b0729a86e454cab75fe54ad45df42d97c8071
  • 可恢复备份:/opt/devbox/devbox.backup-20260723-b71c2c7-pre

本地执行机对 GitHub 443 的一次官方源测试因网络超时失败;相同测试二进制在目标机对官方源通过。PR 保持 open,等待人工 review,不会自动 merge。

@wuxs

wuxs commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator Author

自动发现与安全接管版本已更新并部署(PR head c0dbe5d):

  • make test(race,pkg/apps 71.6%)、go vet ./...、前端 tests 7/7、目标 ESLint、Vite build、make build-all 均通过。
  • 真实 Docker TestTakeoverE2E PASS(48.89s):发现外部 project → 接管 → 编辑/Apply → 重建 controller/worker/repo → restart,始终保留原 project name 与 named volume marker;独立 project 完全 compose down 后不再被发现。
  • Linux 路径安全补充回归:FIFO config path 使用 non-blocking open 并作为非普通文件立即拒绝,不会阻塞接管请求。
  • 已原子替换 192.168.1.2:/opt/devbox/devbox 并重启 devbox.service
  • devbox.service=activeGET /api/v1/health 返回 200;日志显示 Compose runtime enabled
  • 本地/远端 SHA256:eb57002857499405e22a1f96cf6fdf4be63f118bcd6bf9e858e0c13d91473e39
  • 可恢复备份:/opt/devbox/devbox.backup-20260729-c0dbe5d-pre
  • 重启后第一次 health 探测发生在 9092 尚未监听的瞬间,就绪循环随后成功,未触发回滚。

Codex/GPT-5.6-sol xhigh 预 review 因当前 Codex token 401 且 read-only bubblewrap 无法创建 namespace 未完成;限制已写入 PR body。独立 Standards/Spec 自审均无 BLOCKER/HIGH。PR 保持 open,等待人工 review,不会自动 merge。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: 设计并实现 Docker Compose 应用管理

1 participant