ComfyUI Base 3.22.1

ComfyUI Base 3.22.1: verify names every failing test; the sage dry run passes when the release holds the arch (#44)

Two CI fixes, found 2026-10-17.

1. A failing suite's report named 4 of 7 failures

Where	What went wrong	Now
lib/95-summary.sh	pytest ran with -rs, so the short summary
listed only skips	-rfEs: every failed and errored test is named
_build/verify.sh _verify_failures	awk … | head -200 cut the
FAILURES section and broke awk's pipe	every FAILED/ERROR summary
line first, then the sections, capped inside awk (p && n++ < 200)
lib/95-summary.sh	under a package script's set -Eeuo pipefail, a
red suite tripped the base's ERR trap at the pipeline into tee (exit 4,
blaming tee)	`... | tee "$out" && rc="${PIPESTATUS[0]}" ||
rc="${PIPESTATUS[0]}"`

Why not || rc="${PIPESTATUS[0]}" alone: the suite's _bash helper
sources the base without pipefail. There, the pipeline returns tee's 0,
the || never runs, and a red suite reads green. That is the 2.0.14 bug
its test guards against. The && … || … form reads pytest's own status
both ways and never trips the trap. I checked all four cases (pipefail
on and off, pytest 0 and 3) under set -E with an ERR trap.

2. The sage-wheels dry run failed when wheels were already published

A pull request builds every arch (--all) and runs the publish step
dry. The write-once check still ran against the real release and exited
1 on the first arch already there (run 39073460610, sm_100). Now, when
an asset exists, the step compares the published sm_<arch>.sha256 with
the fresh wheel's:

• identical: says so, and drops that wheel from the upload list;
• different, or no sum published (half-published): a ::warning::
  in a dry run, and the run continues;
• different, real publish: exit 1 as before, before anything
  uploads.

The wheels are probably not byte-reproducible (no SOURCE_DATE_EPOCH),
so a PR dry run will usually take the warning path. It no longer fails.

Release

VERSION 3.22.1, handbook line 3 and a §10 entry. package.py --base
rebuilt the zip and MANIFEST.sha256, and --base --check says it is
current.

Tested

Nothing heavy ran on the Mac and no pod was used. On the Mac:

• bash -n and py_compile pass, and the workflow YAML parses.
• The publish step, extracted from the YAML, ran against a fake gh in
  six cases (dry or real × identical, different, half-published). Exit
  codes were 0, 0, 1, 0, 0, 1 as intended.
• The two new tests and the existing verify test that use a Python
  harness passed when called directly.

This PR's GitHub-hosted CI runs the full suite, including the new
errtrace runner test.

🤖 Generated with Claude Code

Co-authored-by: Claude Opus 5.5 noreply@anthropic.com

The zip is what a pod installs. Handbook: comfyui-base-handbook.md

ComfyUI Base 3.22.0

ComfyUI Base 3.22.0: the cuBLAS probe beside the venv build; the install's wall-clock timeline (#42)

Why

Paul's priority 3, install speed. The last Video Replacer pod
(yswh3tkypxhxev, 1x B200 EU-RO-1, base 3.20.0, every row: 42 files,
295.5 GB) took about 5:17 from create to the App on its fresh attempt.
Its 16:55 figure includes a failed first attempt and about 11 minutes
with the pod stopped. Two findings fed this release:

• VENV (116 s) held an apt install of cuda-libraries-dev-13-0 (33
  packages, 2,251 MB fetched one file at a time, 4,146 MiB unpacked). That
  install ran although nothing built from source (PKG_NO_SAGE=1).
• The stage clock has seconds only, so what ran beside what (the
  critical path) was inferred, not read.

What

1. The cuBLAS probe runs beside the venv build.
   _base_cuda_toolchain_start starts in the background next to the models
   and tools jobs. _base_cuda_toolchain_join runs before pkg_post_venv,
   the first place a build can start, so every build still finds the
   toolkit fixed or is told why not. The venv's own install runs before the
   probe in either order. A run that ends early waits for the probe and
   never kills it: an apt stopped halfway leaves dpkg needing dpkg --configure -a. A dry run, runtime mode and BASE_CUDA_FOREGROUND=1
   keep the old order. The clock gains CUDA TOOLCHAIN (in the background, beside the venv build).
2. state/logs/<pkg>_<ts>.timeline.tsv sits beside timings.tsv,
   which is unchanged. Columns are start TAB end TAB seconds TAB what in
   epoch seconds, earliest first. It covers every stage and every
   background stage (SYSTEM, MODELS, TOOLS VENV, CUDA TOOLCHAIN) from its
   start to its end, and every model file twice: download <rel> and
   sha256 <rel> (the re-read at the job's end). Each file's console line
   now ends in (sha256 Ns of it).
3. podctl PODCTL_TIMES=1 stamps each line [HH:MM:SSZ +N.Ns]. It
   is off by default, so parsers read podctl's lines as before.

Tests

suite/test_critical_path.py (unit tier) covers:

• the probe runs in the background, and its verdict, output and warning
  come back at the join;
• the foreground forms (BASE_CUDA_FOREGROUND=1, a dry run);
• a probe that died without a verdict is asked again;
• an early exit waits for the probe and never kills it;
• the pipeline's order;
• the timeline's order and format;
• the per-file spans from the serial path, the parallel path and the
  early pass;
• podctl's stamp.

Nothing ran on the Mac except bash -n, py_compile and package.py --base / --check (house rule: suites run in CI only).

Exercised on

No host. Not run on hardware: the next RunPod pod measures it.

Measurement plan (pod-runner, the D pass)

1. Pin base-v3.22.0 (one more offline dry install). Export
   PODCTL_TIMES=1 for podctl install. install_timings.sh copies
   state/logs/*.timeline.tsv.
2. Use one fresh pod and one clean install. A failed attempt restarts
   the clock at the next create.
3. D runs with BASE_STAGED=1 and PKG_LANES set but empty (the
   orchestrator's yes). Only the two B300-only files (87.2 GB, the 66.3 GB
   long pole of the last pod) arrive after the server.
4. Report:
   • create to App;
   • the critical path, read from timeline.tsv;
   • the VENV seconds and the CUDA TOOLCHAIN row;
   • each file's download and sha256 seconds;
   • the Mac's 46 s, split by the stamped lines.
5. Pass marks: VENV at least 25 s under 116 s on a pod whose image lacks
   cuBLAS; create to App at or under 4:15 on a cached B200.

The ranked levers (GHCR image and network volume not recommended,
nothing paid) and their first-party sources are in the orchestrator's
notes: install-speed-ranked.md and install-speed-sources.md.

🤖 Generated with Claude Code

---

Co-authored-by: Claude Opus 5.5 noreply@anthropic.com

The zip is what a pod installs. Handbook: comfyui-base-handbook.md

ComfyUI Base 3.21.0

ComfyUI Base 3.21.0: pick_lane, the cheapest test GPU that holds a measured peak (#41)

What

py/gpu_fit.py learns which free GPU a smoke should go to.

• LANES, cheapest first: Kaggle T4 (16 GB) and L4 (24 GB), HF ZeroGPU
  large (48 GB) and xlarge (96 GB), Modal's $30 of monthly credits
  from a T4 to a B200, then RunPod (RTX PRO 6000, B200, B300), marked
  paid.
• pick_lane(peak_gib, minutes, needs) returns the first lane whose
  card holds the measured peak plus 10 % and whose own limit holds the
  run:
  • Kaggle: 12 h session
• ZeroGPU: our 30-minute cap, 15 minutes on xlarge since it costs the
  quota twice
• Modal: the minutes $30 buys at the GPU's list price, within Modal's 24
  h per call
• needs takes fast dtypes (fp16 bf16 fp8 fp4, from the compute
  capability by fast_dtypes()), server (ZeroGPU runs per-call
  functions, so it never qualifies) and GPU classes (B200, B300). A
  class is judged by the existing fits(), so GPU_PREFER and
  pick_lane share one rule.
• CLI: python3 py/gpu_fit.py --peak-gib 35.7 --minutes 10 --needs bf16
  prints lane=zerogpu-large ... paid=no. lane=none exits 1, and an
  unknown need exits 2.

Modal prices and the 24 h limit come from modal.com/pricing and Modal's
timeouts guide (read 2026-10-17). They match
ci/modal-smoke/modal_guard.py in the workspace.

Tests

suite/test_unit.py::test_unit_pick_lane_takes_the_cheapest_lane_that_holds_the_peak_and_the_run
covers:

• the lane order and the headroom
• the ZeroGPU and Modal limits
• dtypes, server, B200 through fits()
• the paid fallback, None, and the CLI exit codes

The peaks are the measured ones: Photo Auditor 17.72 GB, Krea 2 bf16
35.7 GB, Flux 2 Dev fp8 about 71 GB.

Nothing ran on the Mac except py_compile, a CLI spot check and
package.py --base --check. This PR's CI runs the suite. No pod used.

After merge: tag base-v3.21.0, then the workspace moves the base
line of repos.txt in its own commit.

🤖 Generated with Claude Code

Co-authored-by: Claude Opus 5.5 noreply@anthropic.com

The zip is what a pod installs. Handbook: comfyui-base-handbook.md

ComfyUI Base 3.20.3

ComfyUI Base 3.20.3: the CPU-quota and launch-env tests hold on any machine (#39)

Workspace integration run 39049424709 (job workspace, on the free
soulcraft-hf runner, base-v3.20.1) still failed three base unit tests.
The code was right; the tests read the runner's own environment:

Test	What the runner did
test_unit_the_cpu_quota_is_the_cgroups_not_the_hosts	2 CPUs
online, so the cap turned the fake 250000 100000 quota into 2, not 3
test_unit_launch_env_precedence_machine_then_packages_then_the_quota
exports OMP_NUM_THREADS=2, which (per the contract) wins over every
package and quota value
test_unit_launch_env_reaches_comfyui_and_only_comfyui	same
OMP_NUM_THREADS=2

Change (suite only; no base code changed)

• Quota test:
• The rounding cases run with _base_cpus_online stubbed to 64, so they
  prove the rounding on any host. With 2 CPUs, rounding up and rounding
  down both give 2, so the old cases proved nothing there.
• The cap is proved twice: with the host's own count (the existing case,
  unchanged), and with a stubbed 2-CPU host (250000 100000 gives 2).
• Launch-env tests:
• A _no_launch_env(monkeypatch) helper deletes every key of the base's
  own BASE_LAUNCH_ENV_KEYS (read from base.sh, so it stays in step
  with the contract) plus any CT2_* from the suite's environment.
• Both tests stub 64 CPUs, so the fake quotas (3 and 2) are never
  capped.
• Every assertion is unchanged, including the one that the machine's own
  OMP_NUM_THREADS=5 wins.
• VERSION 3.20.3 (on top of 3.20.2), handbook line 3 and a §10 entry.
  package.py --base rebuilt the zip (the suite ships), and --base --check says it is current.

Tested

Nothing ran on the Mac. py_compile passes, and I checked by hand that
the edited bash snippets build the intended strings. This PR's CI covers
ubuntu. The soulcraft-hf integration run after the pin move covers the
runner.

🤖 Generated with Claude Code

Co-authored-by: Claude Opus 5.5 noreply@anthropic.com

The zip is what a pod installs. Handbook: comfyui-base-handbook.md

ComfyUI Base 3.20.2

ComfyUI Base 3.20.2: the testbed links the newest copy of a pack two packages vendor (#38)

Why

Workspace integration run
39045535460
failed verify (image-creator-flux-3-klein): ContentGatePrompt and
ContentGateCaption were missing from the testbed. FLUX 3 Klein and the
Video Replacer both vendor ComfyUI-Content-Gate, and testbed.sh
linked each package's copy into custom_nodes by folder name with ln -sfn, so the package walked last (the Replacer, by name order) won
whatever the versions. On a pod the base already keeps the newest copy
(lib/62-own-packs.sh). The testbed now does the same.

Change

• py/own_packs.py: newest <dir>... returns one folder per pack
  name, the newest version of each (numeric compare, so 1.10.0 beats
  1.9.0). A copy with no version loses to one that has a version. On the
  same version the first copy stays, and two different contents under one
  version are named. Notes go to stderr.
• testbed.sh: the derived vendored list goes through newest
  before the link loop (only when there is more than one). --status
  lists each vendored pack's version and the package it comes from.
• suite/test_pkgdirs.py: the helper's rules, plus a --status run
  on a workspace where name order alone would pick the older copy.
• VERSION, handbook record, comfyui-base.zip and MANIFEST.sha256
  rebuilt (--base --check clean).

Not run on the Mac (house rule): this PR's CI is the gate. After the
tag, the workspace's repos.txt pin moves to base-v3.20.2.

🤖 Generated with Claude Code

Co-authored-by: Claude Opus 5.5 noreply@anthropic.com

The zip is what a pod installs. Handbook: comfyui-base-handbook.md

ComfyUI Base 3.20.1

ComfyUI Base 3.20.1: the testbed names a port's holder on Linux without lsof (#37)

Workspace integration run 39045535460 (job `workspace`, on the free
`soulcraft-hf` runner, a Hugging Face CPU Space container) failed two
base unit tests in `suite/test_newest.py`:

`test_unit_testbed_stops_only_the_server_it_started_and_refuses_a_foreign_port`
(line 482): expected `port N is already in use by pid `, got
`pid ?`.

`test_unit_testbed_owned_names_only_its_own_server_and_each_run_folder_gets_its_own`
(line 530): got `is held by pid ?, which this testbed did not start`.

Cause: that container has no `lsof`, and `testbed.sh`'s
`port_holders` answered `?` for any port that accepts a connection
when lsof is missing. GitHub's ubuntu runners have lsof, so the base's
own CI passed.

Change

• `testbed.sh`: `port_holders` tries lsof, then (Linux) a new
  `proc_port_holders`: the LISTEN rows (st `0A`) of `/proc/net/tcp`
  and `tcp6` for the port give socket inodes, and one `ls -l
  /proc/[0-9]*/fd` gives the pids whose fds link those inodes (every
  process of this user, no extra tools, busybox-safe). `?` remains only
  for a port that answers when neither lsof nor /proc can name the holder.
  An lsof that is present but sees nothing now also falls through to
  /proc.
• `suite/test_newest.py`: a `_holder(p, port)` helper decides,
  independently of `testbed.sh`, whether this host can see the listener
  (`lsof -t` names the pid, or `/proc//fd` holds a socket link).
  Both tests demand the pid whenever it can, and accept `?` only when it
  cannot. So on the HF runner, where /proc is readable, they now demand
  the real pid too. The second assertion also gained the trailing comma
  (`held by pid ,`), so a pid prefix no longer matches.
• VERSION 3.20.1, handbook line 3 and a §10 entry; `package.py --base`
  rebuilt the zip and `--base --check` says it is current.

Tested

Nothing ran on the Mac (no local suites). `bash -n testbed.sh` and
`py_compile` pass, and the awk parsing was checked by hand against a
sample `/proc/net/tcp`. This PR's CI is the gate on ubuntu (lsof
path); the soulcraft-hf integration run after the pin move is the check
on the /proc path.

🤖 Generated with Claude Code

Co-authored-by: Claude Opus 5.5 noreply@anthropic.com

The zip is what a pod installs. Handbook: comfyui-base-handbook.md

ComfyUI Base 3.20.0

ComfyUI Base 3.20.0: one ComfyUI per GPU, podctl create --gpus N (#36)

One ComfyUI per GPU, so a multi-GPU pod renders several parts of one job
at once (a clip's shots), each exactly as one GPU would.

• Boot and restart (lib/85-launch.sh, lib/boot.sh): one server
  per GPU, started together. Server 0 is 8188 as before (front, app ports,
  links). Server i is 127.0.0.1:18188+i (8189 is the Video Replacer's
  app port), with CUDA_VISIBLE_DEVICES its own GPU, its own temp folder
  (ComfyUI empties its temp at start) and its own SQLite database; input,
  output and user folders are shared. Every process gets
  COMFY_BASE_SERVERS and COMFY_BASE_SERVER_INDEX; state/servers.list
  names them; the quota's OMP_NUM_THREADS is split by N;
  BASE_SERVERS=1 is the single server. No GPU, or one, is unchanged.
• podctl create --gpus N (1 to 8): count-aware stock check
  (gpuTypes.lowestPrice with gpuCount and dataCenterId, one aliased
  query); every create prints RunPod's price for the pod with RAM and
  vCPUs and refuses when the projected balance covers under one hour of
  today's spend plus this pod. podctl status names the servers.
• GPU check (py/gpu_facts.py, podctl gpu_probe): each card
  judged by its own processes (by uuid).

Read live 2026-10-17: B200 x1 $7.99/h, 188 GB RAM, 24 vCPU; x2 $15.98/h,
502 GB, 48 vCPU (US-CA-2, US-NC-2); x4 and x8 no Secure stock.

Tested against fakes (4-GPU nvidia-smi stub, fake RunPod): the new and
touched tests pass locally; the full gate is this PR's CI. Owed: the GPU
proof on a 2x B200 pod (HANDOFF 7).

🤖 Generated with Claude Code

---

Co-authored-by: Claude Opus 5.5 noreply@anthropic.com

The zip is what a pod installs. Handbook: comfyui-base-handbook.md

ComfyUI Base 3.9.2

ComfyUI Base 3.9.2: fake_pod carries every ZIP_EXTRA path; testbed.sh stops only its own server; no full resolve for a Python torch has no wheels for (#15)

Three harness defects, 2026-10-16. Two came from porting
Soul-Craft/SoulCraft-Workflows#2 (Video Replacer 5.1.0) into
SoulCraft-Org; the third (defect B) from the Video Swapper's pod
install.

1. basetest.fake_pod leaves out shipped directories that are not

packs
It copied a package's top-level files and its node packs (directories
with an __init__.py), so the Video Replacer's app/ (listed in its
ZIP_EXTRA) never reached the fake machine. The Replacer's install test
copied it by hand. fake_pod now also copies every path the script's
ZIP_EXTRA lists, read by the new basetest.zip_extra() the same way
_build/package.py reads it (shlex). A listed file that is missing
raises an error; a file the list leaves out stays off the fake machine,
just as it is off the zip.

2. testbed.sh killed another checkout's ComfyUI

--server and --stop killed whatever was listening on $PORT (lsof).
On a Mac shared by several sessions, that was another checkout's server
on 8199. Now they stop only this checkout's server: the pid recorded in
.testbed-server.pid (now python's own pid), or that pid's child (the
layout of a pid file written by older versions), and only while its
command line is still a ComfyUI main.py. If another process holds the
port, --server exits 1, naming its pid and command and suggesting a
free TESTBED_PORT; --stop leaves it running and says so; --status
says whose server is answering.

3. The venv step tried CPython 3.15 before falling back (defect B)

uv lists 3.15 before torchaudio ships a cp315 wheel. The backend
pre-pick ran torch_pick.py for 3.15 (stepping down through every CUDA
build), then _base_python_pick ran the full derived-set resolve for
3.15, and only then did both fall back to 3.14. The new
_base_family_minors asks uv for the torch family alone (torch,
torchvision, torchaudio) for each minor, newest first, and drops every
minor newer than the first one that resolves; both picks iterate that
list. podctl.py is untouched.

Tests (unit tier)

• test_unit_fake_pod_carries_every_file_zip_extra_ships

test_unit_testbed_stops_only_the_server_it_started_and_refuses_a_foreign_port:
fails against 3.9.1's testbed.sh (the foreign listener gets SIGTERM)

test_unit_a_minor_without_torch_family_wheels_costs_one_small_compile_and_nothing_else:
fails against 3.9.1's 30-venv.sh (3.15 reaches torch_pick and the
full resolve)

Hosts

Exercised on this Mac (macOS, /bin/bash 3.2.57) only:

• the suite;
• testbed.sh, live against another checkout's ComfyUI on 8199:
  refused, named and left running;
• the family check, against the real uv 0.13.1 and index: 3.15 rejected
  in under a second, 3.14 accepted.

Not on a pod. The venv change is tested against a uv stand-in.

The workspace pin follows in SoulCraft-Org/workspace#15.

🤖 Generated with Claude Code

Co-authored-by: Claude Opus 5.5 noreply@anthropic.com

The zip is what a pod installs. Handbook: comfyui-base-handbook.md

ComfyUI Base 3.9.1

ComfyUI Base 3.9.1: podctl create --private works and says its volume is not encrypted; a seed build needs no GPU (#13)

Two defects, both found on 2026-10-16 while setting up the RunPod side.

1. podctl create --private (the default since 3.8.1) always failed,
and claimed encryption it never asked for.
It sent volumeEncrypted: true; REST v1 POST /pods refuses that
("Extra input keys") because the field exists only
on the Pod read-back. Read from RunPod's own sources today: REST v1
PodCreateInput, REST v2 and runpodctl 3.0.1 have
no encryption input; GraphQL's deploy inputs carry only an undocumented
volumeKey string (not used: unverified, and
the docs say bring-your-own-key is not supported); the storage docs name
one way, the console's Encrypt volume box.
Now:

• the body never carries the field;
• a note line says the pod volume will NOT be encrypted, before the
  pod exists, and that at a $0 balance RunPod
  terminates a pod without a network volume, volume and all (billing
  docs);
• the summary prints what the read-back says: ENCRYPTED pod volume,
  pod volume … (not encrypted) or
  (encryption not reported); the delete-on-false is gone;
• the suite's fake RunPod refuses any POST /pods key outside the live
  PodCreateInput, so a field the API does not
  take fails on the Mac first.

2. The template image could never be built. A seed build
(BASE_SEED=1, no GPU) failed its venv verify on
torch.cuda.is_available() is False and rolled back (runpod-template
base-image runs on main, both red). The
install now passes BASE_NO_GPU=1 to py/venv_verify.py for a seed
build only; the CUDA build, the archs and the
closure are still checked, and a real machine with no visible device
still fails.

Docs: hosts/runpod/README.md, _build/pod/README.md, handbook §1 and §10.
Exercised on this Mac against the fake RunPod
and a fake torch (verify.sh green from a clone outside ~/Developer);
not against the live API, and the image build is
CI's to show once the workspace pins this tag.

🤖 Generated with Claude Code

Co-authored-by: Claude Opus 5.5 noreply@anthropic.com

The zip is what a pod installs. Handbook: comfyui-base-handbook.md

ComfyUI Base 3.9.0

ComfyUI Base 3.9.0: ComfyUI, Jupyter Terminal and each customer app are named links

Paul's ask of 2026-10-16: in RunPod's Connect tab, the canvas, each package's app and the terminal are
one-click links with names. RunPod names a port only on a template (GraphQL portsConfig, written by
saveTemplate; REST has no label field and a pod has none of its own), and its proxy links ports, not paths.

• py/app_proxy.py: each app at the root of its own port (:/ -> 127.0.0.1:8188/apps//; ws, upload,
  view, history, queue, interrupt passed through; websockets piped; a "ComfyUI is starting" page while it does
  not answer). The boot starts it after ComfyUI; the install restarts it (boot.sh --apps) from
  state/apps.d/.list, which the BOOT stage writes from a package's APP_PORTS=("||").
• podctl create (RunPod): ports 8188/http "ComfyUI", 8888/http "Jupyter Terminal", one /http per declared app,
  22/tcp, named through a private "comfy-base ports " template that carries the names and nothing else
  (made once per set of names; every field re-sent through saveTemplate, as runpodctl v3 does); a fresh
  JUPYTER_TOKEN per pod; --no-names. deploy --like keeps a names template and gives the clone its own token.
• JupyterLab is never tokenless and never dark: the boot generates a token on any bind; on RunPod's public bind
  it names it once on the container log (the Logs tab), never the boot log.
• podctl status prints the named links (the terminal's with its token); podctl jupyter adds the proxy link;
  podctl tunnel forwards the machine's app ports and prints named localhost links (a --private pod's names).
• Docs: hosts/runpod/README.md "Named ports", handbook rule 9, 1.1, 4 (APP_PORTS), 6, 7, 10; HANDOFF.md.

Exercised on this Mac against fakes only (fake RunPod REST + GraphQL, a raw-socket ComfyUI, fake pods); the
console showing the names is owed on a real pod (HANDOFF.md 1).

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com

The zip is what a pod installs. Handbook: comfyui-base-handbook.md