Panel on distroless: prove it works and measure the delta

[qcentic-adm]

Part of #6

Question

Does the Panel actually build and run on gcr.io/distroless/nodejs24, and what does it cost?

The decision to go distroless is made; this ticket is the evidence that it is achievable, not a re-litigation. Build a throwaway Dockerfile and find out:

• Does better-sqlite3 load? It is compiled in the build stage against glibc; distroless is glibc, so it should — verify rather than assume, including that the prebuilt/node:24-bookworm-compiled .node binary resolves at runtime.
• The current image runs CMD ["node", "bin/panel.mjs"]. Distroless nodejs images have node as the entrypoint, so CMD becomes the script path. Confirm the working invocation.
• HEALTHCHECK currently shells out via CMD node -e "…". With no shell, this needs CMD ["node", "-e", …] exec form, or dropping the in-image healthcheck in favour of a compose-level one. Decide which.
• The image currently does RUN mkdir -p /data && chown node:node /data and USER node. Distroless has no RUN capability at the runtime stage — the directory and its ownership must be staged in the build image and copied. Work out how, and which distroless tag (:nonroot vs default) that implies.
• Measure the CVE delta — before and after, same scanner, so the 74 has a number to be compared against.
• Note anything lost: no shell means no docker exec sh for debugging. Say how an operator is expected to debug the container instead, since that goes in the docs.

Deliverable: a working prototype Dockerfile linked from this issue, the measured CVE delta, and a short list of what the real implementation must not forget.

[qcentic-dev]

Prototype: it works. 192 → 14 CVEs.

Built and run on podman 4.9.3 / linux-arm64, scanned with trivy 0.73.0, vuln DB updated 2026-08-03 07:52 UTC. Everything below is observed output, not inference. Prototype lives on branch wt-10-panel-distroless (commit c19cfd0); ./prototype-distroless.sh reproduces every line of it in one command.

The Dockerfile

# The build stage is untouched from the shipping Dockerfile — full Node on
# bookworm, with the toolchain better-sqlite3's node-gyp build needs. None
# of it reaches the runtime stage, so none of its CVEs ship.
FROM node:24.15.0-bookworm AS build

RUN npm install -g pnpm@11.1.2

WORKDIR /app

COPY package.json pnpm-lock.yaml pnpm-workspace.yaml ./
COPY scripts/require-node-24.mjs scripts/
COPY packages/harness/package.json packages/harness/
COPY packages/panel/package.json packages/panel/
COPY packages/shared/package.json packages/shared/
RUN pnpm install --frozen-lockfile

COPY . .

RUN pnpm --filter @actana/harness build && pnpm --filter @actana/panel build

RUN pnpm --filter @actana/panel deploy --prod --legacy /srv/panel \
  && rm -rf /srv/panel/dist \
  && cp -R packages/panel/dist /srv/panel/dist \
  && cp -R packages/panel/bin /srv/panel/bin

# Distroless has no RUN, so /data cannot be created in the runtime stage.
# Stage the empty directory here and COPY it across. Ownership is the whole
# problem: Docker seeds a fresh named volume from the image's content *and*
# mode at that path, so a root-owned /data yields a volume the Panel cannot
# write to. Measured: COPY resets the *destination* directory to root:root
# 0755 no matter what the staged directory looked like — chown/chmod here is
# discarded. `COPY --chown` below is what actually carries. The mkdir is
# still required; there is nothing to copy otherwise.
RUN mkdir -p /staged/data

# `gcr.io/distroless/nodejs24` is Debian 13 (glibc 2.41), Node 24.18.0, and
# its ENTRYPOINT is ["/nodejs/bin/node"] — so CMD below is argv to node, not
# a command line. The default tag is used deliberately: /etc/passwd already
# carries `nonroot:x:65532:65532` in every variant, so USER below drops
# privilege without needing the `:nonroot` tag.
FROM gcr.io/distroless/nodejs24:latest

ARG IMAGE_SOURCE=https://github.com/actana/control
ARG IMAGE_REVISION=
LABEL org.opencontainers.image.title="Actana Panel" \
      org.opencontainers.image.description="Self-hosted web Panel for driving agentic coding work across your machines. Deploy this one; pair a Core per machine." \
      org.opencontainers.image.source="${IMAGE_SOURCE}" \
      org.opencontainers.image.revision="${IMAGE_REVISION}" \
      org.opencontainers.image.licenses="MIT"

ENV NODE_ENV=production \
    AC_PANEL_DATA_DIR=/data

WORKDIR /srv/panel
COPY --from=build /srv/panel ./

# `--chown=nonroot:nonroot` also works — buildah resolves the name against the
# runtime image's /etc/passwd, which carries nonroot in every distroless
# variant. Numeric anyway: it does not depend on which builder resolves it.
COPY --from=build --chown=65532:65532 /staged/data /data

# Numeric here is load-bearing, not style: Kubernetes' runAsNonRoot admission
# check cannot resolve a username and fails the pod rather than the check.
USER 65532:65532

EXPOSE 7420
VOLUME /data

# Exec form, absolute path. `["node", ...]` does NOT work: HEALTHCHECK argv
# bypasses ENTRYPOINT, and distroless's PATH does not contain /nodejs/bin.
HEALTHCHECK --interval=30s --timeout=5s --start-period=20s \
  CMD ["/nodejs/bin/node", "-e", "const p=process.env.AC_PANEL_PORT??7420;fetch(`http://127.0.0.1:${p}/api/healthz`).then(r=>process.exit(r.ok?0:1),()=>process.exit(1))"]

# Not ["node", "bin/panel.mjs"] — node is the ENTRYPOINT, so CMD is the
# script path alone. Leaving "node" in place makes node try to run a file
# literally named `node`.
CMD ["bin/panel.mjs"]

---

The measured CVE delta

Same scanner, same DB, same invocation both sides:

trivy image --scanners vuln --pkg-types os,library --format json <image>

(no container socket in this environment, so the input was a podman save --format docker-archive tarball via --input; identical findings, only the input differs)

           CRITICAL     HIGH   MEDIUM      LOW  UNKNOWN    TOTAL
baseline os               7       20       65       73        9      174
baseline lib              1        6       10        1        0       18
distroless os             0        0        5        7        0       12
distroless lib            0        1        0        1        0        2
delta                    -8      -25      -70      -66       -9     -178

baseline total   192      (node:24.15.0-bookworm-slim, debian 12.14, 88 OS packages)
distroless total  14      (gcr.io/distroless/nodejs24, debian 13.6, 13 OS packages)

Zero CRITICAL, zero HIGH from the OS. The 12 that remain are libc6 (11) and zlib1g (1) — nothing else in the image has a CVE, because there is nothing else in the image.

The two remaining lang-pkgs findings deserve to be named, because they are the honest part:

	severity	path
@babel/core CVE-2026-49356	LOW	srv/panel/node_modules/…
postcss GHSA-r28c-9q8g-f849	HIGH	srv/panel/node_modules/…

Both are the Panel's own production dependency tree and are byte-identical in both images. The base swap does nothing for them — that is #11's territory, and the one HIGH the distroless image ships is a Panel dependency, not a base-image problem.

Of the 16 library CVEs that did vanish (tar ×6 incl. one CRITICAL, brace-expansion ×4, sigstore/@sigstore/* ×3, picomatch ×2, ip-address), every one lived in usr/local/lib/node_modules/npm/ — npm's own bundled dependencies, shipped by node:24-bookworm-slim and never used at runtime. Distroless has no npm, so they are simply gone. Worth knowing that a meaningful slice of the "before" number was never the Panel's code.

Size: 633 MB → 545 MB. Only −88 MB, because node_modules dominates the image either way. Distroless is a CVE story here, not a size story; don't sell it as one.

Numbers won't line up against the "74" mentioned in #6 — different scanner run, different day, and this counts os,library on the built Panel image rather than a base image. What is comparable is the before/after pair above, which came from one scanner on one afternoon. @#7: the block to match is trivy 0.73.0, vuln DB 2026-08-03 07:52:05 UTC, --scanners vuln --pkg-types os,library.

---

The four questions, answered

1. Does better-sqlite3 load? Yes, fully.

node                   v24.18.0 linux/arm64
glibc                  2.41
native binding         /srv/panel/node_modules/.pnpm/better-sqlite3@12.10.0/node_modules/better-sqlite3/build/Release/better_sqlite3.node
addon dlopen           ok
sqlite :memory:        sqlite 3.53.1 — read back "it loads"
sqlite /data           sqlite 3.53.1 — read back "it loads"

A .node compiled in the bookworm build stage (glibc 2.36, Node 24.15.0) loads on the Debian 13 runtime (glibc 2.41, Node 24.18.0). Forward-compatible in the direction that matters — and note the Node patch versions differ, which is fine only because both are Node 24 (same ABI, NODE_MODULE_VERSION 137).

Not just a synthetic probe — the real Panel, started normally, migrated a real database into the volume:

/data/panel.db         4096b  uid=65532
/data/panel.db-shm    32768b  uid=65532
/data/panel.db-wal    61832b  uid=65532
migrated tables: operator, panel_sessions, cores, core_secrets

and GET /api/healthz → {"ok":true,"status":"ok",…}.

2. The working invocation. CMD ["bin/panel.mjs"] — the script path alone, relative to WORKDIR /srv/panel. Confirmed from inside (process.argv[0] = /nodejs/bin/node) and from a sidecar looking at the container's pid 1:

pid 1: /nodejs/bin/node bin/panel.mjs

3. HEALTHCHECK: keep it, exec form, absolute path. Dropping it in favour of a compose-level check would regress plain docker run and any orchestrator that reads image health, for no gain — the check is three lines. But the naive port is broken in a way that fails silently:

/nodejs/bin/node form -> healthy
["node", ...] form  -> UNHEALTHY

That second line is a healthcheck of ["node", "-e", "process.exit(0)"] — a script that cannot fail — and it still reports unhealthy. HEALTHCHECK argv bypasses ENTRYPOINT, and distroless's PATH (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin) does not contain /nodejs/bin. Verified from inside the image: PATH has node → false.

4. /data, and whether :nonroot is forced. It is not forced. Every distroless variant — default tag included — ships nonroot:x:65532:65532:nonroot:/home/nonroot:/sbin/nologin in /etc/passwd, so USER 65532:65532 on the default tag drops privilege exactly as the :nonroot tag would. Preferring the default tag + explicit USER keeps the security posture stated in our Dockerfile instead of inherited from a tag.

Staging is more subtle than the ticket assumed. Four recipes, measured (Dockerfile.copytest.prototype on the branch):

recipe	result
A. RUN chown in build stage, plain COPY /staged/a /data-a	root:root 0755 — chown discarded
B. as A plus chmod 700	root:root 0755 — mode discarded too
C. COPY --chown=65532:65532 /staged/a /data-c	uid 65532, writable ✓
D. copy the parent, COPY /staged/c-parent /data-d-parent	/data-d-parent/c uid 65532 ✓, parent root ✓

COPY recreates the destination directory itself as root:root 0755 and ignores whatever the source directory's metadata was. Ownership only survives for entries inside a copied tree. So "stage it in the build image and copy it" is necessary but not sufficient — --chown on the COPY is what does the work. The prototype's first build got this wrong and produced /data mode=755 uid=0 gid=0, EACCES on write; that is precisely the bug the real implementation would ship if it staged ownership and trusted it.

This matters because Docker seeds a fresh named volume from the image's content and mode at that path. Get it wrong and every new deployment gets a /data the Panel cannot write to.

---

What is lost, and how an operator debugs instead

docker exec … sh is gone. shell present → NONE — no /bin/sh, no /bin/bash, no busybox. Four replacements, all verified against the running prototype:

1. node is the shell. It is already the entrypoint, so it is always there:

   docker exec <ctr> /nodejs/bin/node -e 'console.log(require("fs").readdirSync("/data"))'
   

   This is enough for filesystem inspection, env dumps, HTTP probes, and opening panel.db with the Panel's own better-sqlite3. Used exactly that way above to list the migrated tables. Absolute path — the PATH trap from question 3 applies to exec too.

2. docker cp needs no shell. Pulled /data/panel.db out of the running container in one command; it is daemon-side.

3. A sidecar in the container's namespaces gets a full shell against the Panel's processes, network and filesystem:

   docker run --rm -it --pid=container:<ctr> --network=container:<ctr> \
     --cap-add=SYS_PTRACE nicolaka/netshoot
   

   Verified: /proc/1/cmdline shows the Panel, ls /proc/1/root/srv/panel reads its filesystem, /proc/1/net/tcp shows it listening on 7420 as uid 65532. This is the answer for anything needing curl, ss, strace or lsof. (Without SYS_PTRACE, namespaces still join but /proc/1/root is denied.)

4. gcr.io/distroless/nodejs24:debug-nonroot ships busybox at /busybox/sh and is otherwise identical. Verified working. A one-line base swap gets an operator a real shell for a session — worth documenting as the escape hatch, not the default.

5. The live debugger works, which is strictly more than the old image offered:

   docker run -e NODE_OPTIONS=--inspect=0.0.0.0:9229 -p 9229:9229 …
   

   Verified: {"Browser": "node.js/v24.18.0", "Protocol-Version": "1.1"}.

---

What the real implementation must not forget

1. COPY --chown on /data, not a staged chown. The staged chown is silently discarded. This is the one that will actually break a deployment.
2. Absolute /nodejs/bin/node in HEALTHCHECK — and anywhere else argv bypasses the entrypoint. ["node", …] fails silently-ish: unhealthy with empty output, no error to grep for.
3. CMD loses the "node" element. ["bin/panel.mjs"], not ["node", "bin/panel.mjs"].
4. USER 65532:65532, numeric. Kubernetes runAsNonRoot cannot resolve a username. The node user (uid 1000) no longer exists — anything referencing it by name breaks.
5. scripts/__tests__/panel-image.test.mjs has four assertions that will fail and each needs a deliberate decision, not a patch-to-green:
   • "builds and runs on the exact Node CI tests against" — asserts every FROM matches node:<ci-node-version>-*. Distroless cannot satisfy this, and the runtime Node version (24.18.0) genuinely differs from CI's (24.15.0). Decide what the invariant becomes.
   • "is multi-stage with a slim runtime" — asserts the last FROM contains -slim.
   • "runs as the unprivileged node user" — asserts the last USER is literally node.
   • "starts the same entry the bare-node path documents" — passes as-is (CMD still contains bin/panel.mjs), but only by luck.
6. Pin the base by digest. gcr.io/distroless/nodejs24 carries 6,980 tags and not one contains a version number — checked the full tag list. There are exactly four mutable names (latest, nonroot, debug, debug-nonroot, plus -amd64/-arm64/-ppc64le/-s390x variants of each); everything else is an opaque 40-hex build SHA. So the only meaningful pin is the digest: today sha256:2ee107907b6dd72e13bc43106822a6148b4835bd4b0e8d8ac7b68ad5d7971251 (Node 24.18.0, Debian 13.6). Whatever Dependabot/Renovate config lands has to update digest pins for a registry with no semver, or the image silently drifts its Node patch version between builds. (amd64 and arm64 are both published, which is what Simplify Actana Control: two images, one installer, three workflows #6 needs.)
7. Build stage and runtime are now on different Debian releases (bookworm 12 vs trixie 13). It works — glibc is backward-compatible in that direction — but it is an unstated dependency. Moving the build stage to node:24-trixie would make it symmetric and is worth considering; do not do it accidentally in the other direction.
8. Verify HEALTHCHECK survives the CI builder. Podman silently drops it unless --format docker; this prototype needed the flag. Confirm docker/build-push-action with multi-platform output preserves it, or the published image ships without a healthcheck while local builds look fine.
9. scripts/smoke-panel-image.mjs is safe — checked; it only uses docker run / logs / cp-free operations and never shells into the container. No change needed, but re-check if the smoke test grows.
10. gcr.io is a third registry in the build path. Simplify Actana Control: two images, one installer, three workflows #6 settles Docker Hub primary / GHCR mirror for publishing; pulling the base from Google's registry is a separate availability and rate-limit dependency on every build.
11. Docs owe operators the debugging section above. "No shell" is a support burden if it is discovered during an incident rather than read beforehand.

Leaving this open — the prototype is evidence, the decisions still need writing up into the handoff spec.

[qcentic-dev]

Closed — outcome recorded on #6. Decisions locked in ADR 0016 D20–D26; the work is #42, #43, #44, #60.

The prototype is the evidence for the whole Panel decision and every number in D20 is yours: 192 → 14, OS 174 → 12, library 18 → 2, zero CRITICAL and zero HIGH from the OS.

Two things this ticket settled beyond its brief:

• It closed Simplify Actana Control: two images, one installer, three workflows #6's rootless open item (D21). :nonroot is not forced — every variant already carries nonroot:x:65532:65532, so numeric USER 65532:65532 on the default tag keeps the posture stated in our Dockerfile. Numeric because Kubernetes' runAsNonRoot cannot resolve a username.
• It found the failure mode that would have shipped (D22). The staged chown is silently discarded — COPY recreates the destination as root:root 0755 regardless — so "stage it and trust it" gives every new deployment a /data the Panel cannot write to. Recipes A–D are in the ADR as a named clause, not a footnote.

Your item 11 — "docs owe operators the debugging section" — had no destination in #15's tree (G1). It has one now: #60, docs/operate/troubleshooting.md, which has to tell two opposite stories on one page (Panel: no shell at all; Core: docker exec core bash just works). All five verified techniques carry, including the SYS_PTRACE caveat on the netshoot sidecar.

Item 7 is decided rather than left as "worth considering": the build stage moves to trixie so both stages are Debian 13 (D25), with bookworm acceptable only if the build fails and the asymmetry gets a comment naming its direction. Item 8 — verify the healthcheck survives the CI builder — is a done-condition on #43, because the failure is silent.