Docker Hub publishing has been failing since March — `nodeodm:gpu` frozen at 2025-10-22, GPU SIFT silently broken

[erickeller]

Summary

None of the three Publish Docker* workflows have succeeded since April. opendronemap/nodeodm:gpu has therefore not been rebuilt since 2025-10-22, while its own base image opendronemap/odm:gpu was rebuilt 2026-08-04. Users pulling the current tag get a ~10-month-old image, and on it GPU SIFT cannot work at all — silently, with no error in the task log.

CI status

Workflow	File	Last success	Latest runs
Publish Docker	.github/workflows/publish-docker.yaml	2026-03-25	failure 2026-04-30, 2026-04-28
Publish Docker GPU	.github/workflows/publish-docker-gpu.yaml	(before 2026-03-13)	failure 2026-04-30, 04-28, 03-25, 03-17, 03-17, 03-13
Publish Docker Intel GPU	.github/workflows/publish-docker-gpu-intel.yaml	2026-03-17	failure 2026-04-30, 2026-04-28

Two distinct failure modes on the GPU workflow:

• 2026-03-13 → 2026-03-25 — failed at step Build and push Docker image (includes two manual workflow_dispatch retries)
• 2026-04-28, 2026-04-30 — failed at step Login to DockerHub, which looks like expired/missing Docker Hub credentials

There have been no workflow runs at all since 2026-04-30.

• https://github.com/OpenDroneMap/NodeODM/actions/runs/25143350546 (GPU, 2026-04-30)
• https://github.com/OpenDroneMap/NodeODM/actions/runs/25031580143 (GPU, 2026-04-28)
• https://github.com/OpenDroneMap/NodeODM/actions/runs/25143350540 (CPU, 2026-04-30)

Docker Hub tag dates confirm the freeze:

tag	last_updated
opendronemap/odm:gpu (base image)	2026-08-04
opendronemap/nodeodm:latest / master	2026-03-25
opendronemap/nodeodm:gpu.intel	2026-03-17
opendronemap/nodeodm:gpu	2025-10-22

The last successful gpu publish was the hotfix for #252. Since then every attempt has failed. #174 raised a related rebuild-cadence problem.

Impact: GPU SIFT is silently disabled in the published image

In the current opendronemap/nodeodm:gpu (sha256:214fe6a4421fe5283648400e9ce455457bb866c87a20a325fddfd494e4ad482a), the CUDA 11 runtime is missing, so opensfm's pypopsift can never load:

$ ls -la /usr/local/cuda-11.8/targets/x86_64-linux/lib/ | grep cudart
lrwxrwxrwx 1 root root      17 Sep 21  2022 libcudart.so -> libcudart.so.11.0   # dangling
-rw-r--r-- 1 root root 1198880 Sep 21  2022 libcudart_static.a

$ ldd /code/SuperBuild/install/bin/opensfm/opensfm/pypopsift.cpython-38-x86_64-linux-gnu.so | grep cuda
        libcudart.so.11.0 => not found

$ dpkg -l | grep -i cuda | awk '{print $2, $3}'
cuda-toolkit-11-8-config-common 11.8.89-1
cuda-toolkit-11-config-common 11.8.89-1
cuda-toolkit-config-common 12.3.52-1
libcufile-11-8 1.4.0.31-1
libcusolver-11-8 11.4.1.48-1
libnccl2 2.16.2-1+cuda11.8
# no cuda-cudart package at all

Result, from inside the running container:

$ docker exec -w /code webodm-node-odm-1 python3 -c \
    "from opendm.gpu import get_cuda_compute_version, has_popsift_and_can_handle_texsize as f; \
     print(get_cuda_compute_version(0)); print(f(4032,3024))"
(8, 9)
[WARNING] libcudart.so.11.0: cannot open shared object file: No such file or directory
False

has_popsift_and_can_handle_texsize() swallows this as a warning, so opendm/osfm.py quietly falls back to CPU feature extraction. Nothing in the task log indicates a problem — the job simply runs several times slower. Searching the org for libcudart returns 0 issues, which I suspect is because the failure is invisible rather than because it is rare.

This appears to be collateral from the fix for the CUDA 11.2 conflict discussed in
https://community.opendronemap.org/t/opendronemap-nodeodm-gpu-nvidia-smi-detected-cannot-initialize-cuda/21124
— cuda-cudart-11-2 was removed and nothing installed cuda-cudart-11-8 in its place. A rebuild
on the current opendronemap/odm:gpu (CUDA 12.9 / Ubuntu 24.04) should resolve it, together with
ODM PR #2060 ("Restore DEPENDS opensfm for PyPopsift GPU module install order", merged 2026-08-04).

Minor, related

Dockerfile.gpu on master still contains:

RUN echo /usr/local/cuda-11.2/compat >> /etc/ld.so.conf.d/989_cuda-11.conf && ldconfig

/usr/local/cuda-11.2/compat does not exist in the built image — it leaves a dangling entry in
/etc/ld.so.conf.d/. Probably obsolete now that the base is on CUDA 12.9.

Suggested actions

1. Refresh the Docker Hub credentials used by the Publish Docker* workflows.
2. Re-run publish-docker-gpu.yaml and confirm the March Build and push Docker image failure is resolved by the newer opendronemap/odm:gpu base.
3. Consider a scheduled rebuild, or a trigger on base-image updates, so nodeodm:gpu cannot drift ~10 months behind odm:gpu again.
4. Drop the cuda-11.2/compat line from Dockerfile.gpu.

Environment

• Host: Ubuntu, NVIDIA RTX 4060 Ti 16 GB, driver 580.173.02 (CUDA 13.0), compute capability 8.9
• WebODM 3.0.1, opendronemap/nodeodm:gpu @ sha256:214fe6a4… (ODM 3.5.6)
• nvidia-container-toolkit 1.19.1; GPU is correctly passed into the container (nvidia-smi works inside, OpenMVS densification does use CUDA)

[erickeller]

Dockerfile.gpu-nodeodm-fixed.txt

Three follow-ups since filing

1. The March Build and push Docker image failures

Dockerfile.gpu on master runs the dependency script before the source is copied into the image:

RUN bash install_deps.sh && \        # <- runs here
    ln -s /code/SuperBuild/install/bin/untwine /usr/bin/untwine && \
    ...
RUN mkdir /var/www
WORKDIR "/var/www"
RUN useradd -m -d "/home/odm" -s /bin/bash odm
COPY --chown=odm:odm . /var/www      # <- but the file only arrives here

Building from a clean checkout fails immediately:

#6 0.176 bash: install_deps.sh: No such file or directory
ERROR: failed to solve: process "/bin/sh -c bash install_deps.sh && ..."
       did not complete successfully: exit code: 127

That matches the Build and push Docker image step failing on 2026-03-13, 03-17 (x2) and 03-25.

The fix is to move the mkdir / WORKDIR / useradd / COPY block ahead of it:

-RUN bash install_deps.sh && \
-    ln -s /code/SuperBuild/install/bin/untwine /usr/bin/untwine && \
-    ln -s /code/SuperBuild/install/bin/entwine /usr/bin/entwine && \
-    ln -s /code/SuperBuild/install/bin/pdal /usr/bin/pdal && \
-    ln -s /var/www/node.sh /usr/bin/node
-
 RUN mkdir /var/www
 WORKDIR "/var/www"
 RUN useradd -m -d "/home/odm" -s /bin/bash odm
 COPY --chown=odm:odm . /var/www
+
+RUN bash install_deps.sh && \
+    ln -s /code/SuperBuild/install/bin/untwine /usr/bin/untwine && \
+    ln -s /code/SuperBuild/install/bin/entwine /usr/bin/entwine && \
+    ln -s /code/SuperBuild/install/bin/pdal /usr/bin/pdal && \
+    ln -s /var/www/node.sh /usr/bin/node

There is a second failure immediately after it: node.sh wraps node but not npm, and nvm is
not on PATH in a fresh layer, so the following bare RUN npm install --production also exits
127. Sourcing nvm explicitly fixes it:

RUN bash -c 'source $NVM_DIR/nvm.sh && nvm use $NODE_VERSION && \
             npm install --production && mkdir -p tmp && node index.js --powercycle'

2. With those two changes the GPU image builds and PopSift works

docker build -f Dockerfile.gpu -t nodeodm:gpu-local . then succeeds against the current
opendronemap/odm:gpu base (ODM 3.6.1, CUDA 12.9), tested on an RTX 5070 Ti (sm_120):

$ docker run --rm --gpus all -w /code --entrypoint python3 nodeodm:gpu-local -c \
    "from opendm.gpu import has_popsift_and_can_handle_texsize as f; print(f(4032,3024))"
[INFO]    popsift can handle texture size 4032x3024
True

True, against False + libcudart.so.11.0: cannot open shared object file on the published
gpu image. So a rebuild on the current base resolves the silent CPU fallback in the original
report by itself — no separate cuda-cudart patch needed, because the CUDA 12.9 base ships a
real libcudart.so.12.

While there: the RUN echo /usr/local/cuda-11.2/compat >> /etc/ld.so.conf.d/989_cuda-11.conf
line can be dropped. That directory does not exist in the current base and only leaves a dangling
entry in /etc/ld.so.conf.d/.

3. The April failures look like credentials, not code

The two most recent runs failed at the Login to DockerHub step (2026-04-28,
2026-04-30) rather than at
build. Both postdate the WebODM fork announced on 6 April 2026
(webodm.org/blog/announcement), which stated that "If
you use ODM or NodeODM, you can find updated forks at ODX and NodeODX, for which stable docker
images will continue to receive updates and improvements at webodm/odx and webodm/nodeodx."

Given that timing, it seems worth checking whether the DOCKERHUB_* secret in this repo still
belongs to an account holding push rights to opendronemap/nodeodm. I have no visibility into
the org's credentials — this is only an inference from the failure step and the dates.

4. For reference, the fork already carries both fixes

WebODM/NodeODX/Dockerfile.gpu
(forked from this repo per the announcement above) already has COPY . /var/www ahead of
install_deps.sh and has removed the cuda-11.2/compat line, so the changes in §1 can be ported
rather than written from scratch.

Correction to my original report

I implied the gpu tag had simply been neglected. Having since learned of the April 2026 fork, I
want to be explicit that I am not reporting this as a deprecation: opendronemap/nodeodm:gpu
is still what --gpus-enabled installs on this stack pull today, and on it GPU feature extraction
cannot work at all. That is the whole of my report — I have no view on the fork itself and am
raising this purely as a packaging/CI issue.