Proposal: Runtime images — bring your own image for a computing unit

[aicam]

Summary

A runtime image is the image a computing unit runs, built from a Dockerfile the user owns.
A computing unit started from one runs that image instead of the deployment's default, so a
workflow can use interpreters, system packages and compiled libraries the default image does
not have.

We have a working prototype and would like opinions before it goes any further — especially
on the questions at the end.

This is the same problem area as #7475, approached from the computing unit rather than the
operator.

The problem

Today a user's only way to add a dependency is a Python virtual environment: a list of pip
specs installed into a venv on the computing-unit image's own interpreter. Three common needs
fall outside that definition, and no amount of effort within the mechanism reaches them:

Need	Why a venv cannot	An image can
A different Python version	venv is built from the image's interpreter	FROM a different base
A system package (hmmer, ffmpeg, a driver)	a venv holds Python packages only	apt-get install as root, at build time
A package that compiles from source	there is no build step, only pip install <spec>	any RUN you like

A shell on a running pod does not close the gap either. That shell is the unprivileged
texera user, so apt install fails, and anything it did install would live in the
container's writable layer and vanish on the next restart.

The case that forced it: AlphaFold 3

AlphaFold 3 is a sharp example because it
fails a venv in all three ways at once — it needs Python ≥ 3.12 (the engine's is 3.10),
it is not on PyPI and compiles a C++ extension, and it calls jackhmmer/nhmmer,
which are executables, not Python packages.

This is not a hypothetical dependency. It is the shape of most scientific tooling our users
bring, and each row above is a hard stop rather than an inconvenience.

The prototype

Runtime Images in the sidebar → New runtime image → the editor opens pre-filled with
the computing-unit image's own Dockerfile, so the starting point is what already exists rather
than a blank file. Saving starts a build; Logs is readable during it and after. When
creating a computing unit, the runtime image is picked from a dropdown — only READY ones
appear.

flowchart LR
  D["user Dockerfile"] --> J["BuildKit Job<br/>(rootless)"]
  J -- "push &lt;registry&gt;/&lt;id&gt;:&lt;n&gt;" --> R["in-cluster registry"]
  R -- "kubelet pulls" --> P["computing unit pod"]

Loading

Four decisions worth commenting on:

• BuildKit, rootless. Kaniko was archived in June 2025. Rootless because a build runs
  arbitrary user-supplied instructions, and a privileged builder would make the cluster's
  isolation depend on the Dockerfile being well behaved.
• The tag carries a build number. A rebuild publishes :n+1 rather than overwriting
  :n, so a computing unit already running an earlier build is untouched.
• Status is reconciled on read. A build finishes on the cluster, not in the service, so a
  row learns its outcome when someone lists or opens it — no background threads, no leader
  election.
• Nothing is migrated. Python virtual environments still exist and still work for per-UDF
  selection.

It runs

AlphaFold 3's data pipeline, on minikube, 2 CPU / 4 GiB, no GPU — AF3 3.0.5 built from
source, calling the real jackhmmer through alphafold3.data.tools.jackhmmer, on Python
3.12.14, from an engine whose own interpreter is 3.10:

protein	residues	msa_depth	search_seconds
ubiquitin	76	466	0.075
NEDD8	81	459	0.274
SUMO1	101	322	0.297

Flipping the UDF back to the default Python environment fails the identical workflow with
No module named 'alphafold3', which is the point — the selection is load-bearing, and what
it selects is something no venv could have built.

(Structure prediction is out of scope here: no GPU, no model parameters, no 630 GB databases.
This is the dependency problem, not an AF3 port.)

Known limits of the prototype

• The registry is plain HTTP with no authentication; it works only because container
  runtimes treat the Service CIDR as insecure.
• Nothing constrains what a Dockerfile may do. An image that omits USER texera runs as
  root in the pod, because the pod spec pins no securityContext of its own.
• No quota on how many runtime images a user creates, how large one may be, or how long
  old tags are kept.
• Deleting a runtime image leaves its pushed layers in the registry.

Questions

1. Is the computing unit the right granularity? Running an operator inside a container #7475 asks for a container per operator.
   Per-unit is much cheaper to build and reason about, but one workflow then cannot mix two
   incompatible dependency sets. Is that limit acceptable, or is per-operator the real target?
2. Should users be able to name an existing image (one on Docker Hub, or any bare image
   reference) instead of authoring a Dockerfile we build? That skips the builder entirely, at
   the cost of any control over what is inside.
3. Who is allowed to create one? A build is arbitrary code execution with network access
   on the cluster. Admins only, an allowlist, or every user with quotas?
4. Do Python virtual environments stay? They are strictly weaker, but simpler for the
   common "just add a pip package" case, and they select per-UDF rather than per-unit.
5. Sharing. Runtime images are currently private to their owner. Should they be shareable
   like datasets and workflows, or published to a curated library?

[chenlica]

This proposal was based on several offline discussions between @aicam @tanishqgandhi1908 and myself, motivated a real use case of Alpha Fold 3.

This proposal includes 1) how to build and maintain an image; and 2) how to use an image for a CU. The first task is what tools such as DockerHub do, so we should avoid inventing the wheels. So I believe we should focus on 2). We can just let a user specify a URL to a DockerHub image when creating a CU, without considering how it's created.

The following are my corresponding thoughts on those questions:

1. Yes

2. Not a focus of this work

3. Not a focus of this work

4. PVEs co-exist with this feature.

5. Not a focus of this work