node:22-alpine builds fail with "alpine-baselayout: failed to commit var/run" when using Kaniko on Kubernetes

[onurridvanoglu]

Environment

• Platform: linux/amd64 (GKE Kubernetes nodes, running Kaniko builds)
• Docker Version: Kaniko executor (v1.x, running as a Kubernetes pod in GitLab CI)
• Node.js Version: 22.x / 20.x
• Image Tag: node:22-alpine (Alpine 3.23), node:20-alpine (Alpine 3.23)

Expected Behavior

Running apk update && apk upgrade in a Dockerfile based on node:22-alpine should complete successfully, as it did with previous Alpine versions (3.20, 3.21, 3.22). This is a common pattern in Dockerfiles to ensure the base image has the latest security patches before installing additional packages.

  FROM node:22-alpine
  RUN apk update && apk upgrade && \
      apk add --no-cache python3 make g++ gcc libc6-compat    

Current Behavior

When building the above Dockerfile with Kaniko inside a Kubernetes pod, apk upgrade fails during the alpine-baselayout package upgrade (3.7.1-r8 → 3.7.2-r0):

 (1/3) Upgrading alpine-baselayout-data (3.7.1-r8 -> 3.7.2-r0)
 (2/3) Upgrading alpine-baselayout (3.7.1-r8 -> 3.7.2-r0)                                                                                                 
   Executing alpine-baselayout-3.7.2-r0.pre-upgrade                                                                                                       
   * rm: can't remove 'var/run/secrets/kubernetes.io/serviceaccount/token': Read-only file system                                                         
   * rm: can't remove 'var/run/secrets/kubernetes.io/serviceaccount/namespace': Read-only file system                                                     
   * rm: can't remove 'var/run/secrets/kubernetes.io/serviceaccount/ca.crt': Read-only file system
   * rm: can't remove 'var/run/secrets/kubernetes.io/serviceaccount/..data': Read-only file system                                                        
   * rm: can't remove 'var/run/secrets/kubernetes.io/serviceaccount/..2026_03_27_08_27_57.1948645492/ca.crt': Read-only file system
   * rm: can't remove 'var/run/secrets/kubernetes.io/serviceaccount/..2026_03_27_08_27_57.1948645492/token': Read-only file system                        
   * rm: can't remove 'var/run/secrets/kubernetes.io/serviceaccount/..2026_03_27_08_27_57.1948645492/namespace': Read-only file system                    
 ERROR: alpine-baselayout-3.7.2-r0: failed to commit var/run: Is a directory                                                                              
 1 error; 23.2 MiB in 31 packages                                                                                                                         
 error building image: error building stage: failed to execute command: waiting for process to exit: exit status
1

The alpine-baselayout 3.7.2 upgrade script attempts to restructure /var/run (converting it from a directory to a symlink to /run). In a Kubernetes environment, /var/run/secrets/kubernetes.io/serviceaccount/ is a read-only projected volume mount. Kaniko runs unprivileged and without a full container runtime, so the build process sees the host pod's filesystem mounts. The upgrade script cannot remove or modify these read-only mounted files, causing the entire build to fail.

Possible Solution

Since the Node.js Alpine images are based on the upstream Alpine base image, there are a few possible approaches:

1. Pin the Alpine version in the image tags: The node:22-alpine tag currently resolves to Alpine 3.23, which includes the problematic alpine-baselayout upgrade path. Keeping Alpine 3.22 as the default (or providing clearer documentation about this breaking change) would prevent unexpected build failures.
2. Document the Kaniko/Kubernetes incompatibility: At minimum, the known issue with apk upgrade in Kaniko-based builds on Kubernetes should be documented, so users are aware that node:XX-alpine images based on Alpine 3.23 may break in these environments.
3. Upstream fix in Alpine: This is ultimately an Alpine packaging issue where alpine-baselayout 3.7.2's pre-upgrade script doesn't gracefully handle read-only filesystem mounts. However, since the Node.js image inherits this behavior, users encounter it here first.

Steps to Reproduce

1. Set up a Kubernetes cluster with a GitLab CI runner (or any CI system that uses Kaniko for in-cluster Docker builds)
2. Create a Dockerfile:

FROM node:22-alpine                                                                                                                                      
RUN apk update && apk upgrade && \
    apk add --no-cache python3 make g++ gcc libc6-compat

3. Build the image using Kaniko inside a Kubernetes pod:
   /kaniko/executor --context . --dockerfile Dockerfile --destination <registry>/test:latest --cache=true
4. The build fails with the alpine-baselayout error shown above.

Workaround: Pin to an older Alpine version:
FROM node:22-alpine3.22
or remove apk upgrade from the Dockerfile:
RUN apk add --no-cache python3 make g++ gcc libc6-compat

Additional Information

• This issue affects any node:XX-alpine image that resolves to Alpine 3.23 when built with Kaniko on Kubernetes. Both node:20-alpine and node:22-alpine are affected.
• Pinning to node:22-alpine3.22 or node:20-alpine3.20 resolves the issue immediately.

[MikeMcC399]

Have you tried using gcompat instead of libc6-compat? (I suspect however this will not help.)

References:

• https://github.com/nodejs/docker-node/blob/main/README.md#nodealpine
• https://wiki.alpinelinux.org/wiki/Software_management

[MikeMcC399]

Regarding your suggestion to pin node:22-alpine to node:22-alpine3.22

node:22-alpine is a floating tag that is expected to move to point to the latest Node.js 22 release and the latest Alpine release. Currently it's equivalent to node:22.22.2-alpine3.23 and that can be expected to be bumped to node:22.23.0-alpine3.23 soon. https://alpinelinux.org/releases/ describes how they release in May and November, so then it will float again to node:22.xx.xx-alpine3.24.

The tag node:22-alpine3.22 is also a floating tag, which floats on the Node.js 22 version and is pinned to alpine3.22.

Since pinning to node:22-alpine3.22 works around your issue, that would be the one to use for you, at least for the moment.

node:20-alpine3.20 as an alternative would have a short support lifetime, as Alpine Linux 3.20 enters end-of-support in a few days on 2026-04-01 followed by Node.js which enters end-of-life on 2026-04-30.

If using gcompat doesn't solve the issue then you will probably need to take it to Alpine to look at.

[yosifkit]

This seems like a bug with Kaniko, but it is an archived project with no active forks, so it is unlikely to get any fixes there. I don't think it is important for the node image (or Alpine) to work around, or note incompatibilities with, an abandoned project that builds containers in a non-standard way.