Skip to content

fix(docker): SLS-377 install torchvision in GPU worker image#101

Merged
deanq merged 1 commit into
mainfrom
deanq/sls-377-worker-torchvision
Jul 10, 2026
Merged

fix(docker): SLS-377 install torchvision in GPU worker image#101
deanq merged 1 commit into
mainfrom
deanq/sls-377-worker-torchvision

Conversation

@deanq

@deanq deanq commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Problem

A deployed Flash app importing torchvision fails at worker runtime with ModuleNotFoundError (customer escalation SLS-377, Zendesk #41953).

Root cause is a three-part gap:

  1. flash build/deploy unconditionally strips torchvision from the deployment tarball — SIZE_PROHIBITIVE_PACKAGES = {torch, torchvision, torchaudio, triton}, merged as a set union, so there is no CLI flag to force-include it. The justification is that these are "already provided by the GPU base images (runpod/pytorch:*)".
  2. That assumption is false for torchvision. Probing the current base image (runpod/pytorch:1.0.3-cu1281-torch291-ubuntu2204): torch, torchaudio, triton are present — torchvision is not.
  3. The worker Dockerfile re-adds numpy explicitly (it hit the same gap) but had no torchvision backfill.

Net: stripped from tarball -> absent from base image -> never reinstalled -> import fails at runtime.

Fix

Install torchvision in the GPU worker image, mirroring the existing numpy backfill:

  • Pin TORCHVISION_VERSION=0.24.1, the release paired with the base torch 2.9.1 (bump both together on a torch upgrade).
  • Install from the CUDA wheel index to stay CUDA-aligned.
  • Add a build-time import torchvision assertion so a future torch bump that breaks the pairing fails the build, not the customer.

Not fixed by un-excluding torchvision from the tarball: its wheels are CUDA/platform-specific and depend on torch (which stays excluded), so bundling risks a CUDA mismatch and a second torch. The Dockerfile is the correct layer, consistent with the numpy precedent.

Test plan

  • docker buildx build --platform linux/amd64 --build-arg PYTHON_VERSION=3.12 . — succeeds; torchvision resolves to 0.24.1+cu128 with torch==2.9.1 already satisfied (no torch re-pull).
  • Build-time verify stage prints: Python 3.12 OK, torch 2.9.1+cu128, torchvision 0.24.1+cu128, numpy 2.5.1.
  • make quality-check passes (14/14 handler tests + lint/format/typecheck/coverage).

Follow-ups (out of scope here)

  • CPU images have the same gap — the tarball exclusion is unconditional, so a CPU Flash app importing torchvision would also fail. Left for a separate change/ticket since CPU torchvision is a different wheel and adds image weight.
  • The SIZE_PROHIBITIVE_PACKAGES comment in flash build.py ("already provided by the GPU base images") is now inaccurate for torchvision and should be corrected in a flash-side PR.

The runpod/pytorch base image ships torch/torchaudio/triton but not
torchvision, and `flash build` auto-excludes torchvision from the
deployment tarball (SIZE_PROHIBITIVE_PACKAGES) on the assumption the
base image provides it. Nothing backfilled it, so any deployed Flash
app importing torchvision failed at runtime with ModuleNotFoundError.

Install torchvision in the worker image, mirroring the existing numpy
backfill: pin to the release paired with the base torch (2.9.1 ->
0.24.1) and use the CUDA wheel index to stay CUDA-aligned. Add a build
-time import assertion so a future torch bump that breaks the pairing
fails the build instead of the customer.

Fixes SLS-377.
@deanq deanq changed the title fix(docker): install torchvision in GPU worker image (SLS-377) fix(docker): SLS-377 install torchvision in GPU worker image Jul 10, 2026
@deanq deanq requested review from KAJdev, Copilot and jhcipar July 10, 2026 02:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Installs torchvision into the GPU worker Docker image to prevent runtime ModuleNotFoundError when Flash build/deploy strips torchvision from the deployment tarball and the runpod/pytorch:* base image does not provide it.

Changes:

  • Add a pinned TORCHVISION_VERSION build arg with rationale and upgrade guidance alongside TORCH_VERSION.
  • Install torchvision in the worker image from the CUDA wheel index (TORCH_INDEX_URL) to match the base image’s CUDA/torch build.
  • Extend the build-time verification step to import and print the installed torchvision version.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@deanq deanq merged commit 53aeb8a into main Jul 10, 2026
26 checks passed
@deanq deanq deleted the deanq/sls-377-worker-torchvision branch July 10, 2026 22:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants