Skip to content

Releases: ussoewwin/A1111-for-Python3.14

Release list

v3.0.4 - Early NumPy Validation & Self-Healing against False Torch GPU Startup Crash

v3.0.4 - Early NumPy Validation & Self-Healing against False Torch GPU Startup Crash Latest
Latest

Choose a tag to compare

View all tags
@ussoewwin ussoewwin released this 13 Oct 16:07
v3.0.4
23b7b87

① Overview of the NumPy Early Validation & Startup Deadlock Fix

1. Background (State before the change)

In Python 3.14 environments, launching A1111 via webui-user.bat could abruptly abort during startup with the following error:

RuntimeError: Torch is not able to use GPU; add --skip-torch-cuda-test to COMMANDLINE_ARGS variable to disable this check

Despite the error message pointing to GPU or CUDA issues, the underlying GPU hardware, NVIDIA drivers, and PyTorch (torch==2.14.0+cu132) installation were completely operational.

2. Root Cause Analysis (Technical Deep Dive)

  1. Destructive Unconditional Reinstallation on Every Launch:
    Previously in modules/launch_utils.py, prepare_environment() unconditionally ran:

    • pip uninstall numpy -y followed by pip install --no-cache-dir numpy==2.5.3
    • pip uninstall scipy -y followed by pip install --no-cache-dir scipy==1.16.1
    • A secondary --force-reinstall --no-deps --no-cache-dir numpy==2.5.3 after extension installers.
      If any startup process was closed, killed, or interrupted while running these uninstalls, the venv/Lib/site-packages/numpy directory was left in a half-uninstalled state (e.g. version.py or _distributor_init.py remained while __init__.py and _core/ were removed).

  2. PEP 420 Namespace Package Trapping PyTorch Imports:
    Under Python 3.3+ (PEP 420), any directory without an __init__.py is resolved by CPython as an implicit Namespace Package (<module 'numpy' (namespace)>).
    When launch.py executed its startup check import torch; assert torch.cuda.is_available():

    • torch/__init__.py imports torch.testing (line 2720), which imports torch.testing._comparison.
    • torch.testing._comparison contains: 
      try:
    import numpy as np
    HAS_NUMPY = True
except ModuleNotFoundError:
    HAS_NUMPY = False
if HAS_NUMPY:
    _IS_DATACLASS_SKIP_TYPES = (*_IS_DATACLASS_SKIP_TYPES, np.ndarray, np.generic)
    • Because site-packages/numpy existed without __init__.py, import numpy as np did not raise ModuleNotFoundError and succeeded as a namespace package.
    • Accessing np.ndarray immediately threw AttributeError: module 'numpy' has no attribute 'ndarray'.
    • Furthermore, torchvision.utils unconditionally executes import numpy as np, which also crashed with ModuleNotFoundError / AttributeError.

  3. Startup Ordering Deadlock:
    In modules/launch_utils.py, the check_run_python("import torch; assert torch.cuda.is_available()") check was positioned before line 735 (where numpy was reinstalled).
    Because check_run_python captures standard error and only checks the process exit code, the AttributeError: module 'numpy' has no attribute 'ndarray' from import torch caused the GPU check to return non-zero, immediately raising RuntimeError: Torch is not able to use GPU. As a result, execution aborted before it could ever reach the NumPy reinstallation block, creating a permanent startup deadlock.

3. What Was Changed

  1. Pre-Check Self-Healing & Early Validation:

    • Added an early validation step in modules/launch_utils.py before _apply_torch_stack_pip_constraint() and the torch.cuda.is_available() test.
    • It verifies whether NumPy is cleanly importable: check_run_python("import numpy; assert hasattr(numpy, 'ndarray') and numpy.__version__ == '2.5.3'").
    • If missing or corrupted, any broken site-packages/numpy residue directory is safely purged via shutil.rmtree() to prevent PEP 420 namespace package poisoning, followed by a clean reinstall of numpy==2.5.3.

  2. Elimination of Redundant Startup Reinstall Cycles:

    • Guarded the startup numpy==2.5.3 and scipy==1.16.1 installation blocks with check_run_python assertions.
    • If numpy==2.5.3 (with valid ndarray) and scipy==1.16.1 are already present in the environment, the destructive pip uninstall and --no-cache-dir network downloads are skipped entirely.
    • The post-extension re-pin is similarly guarded to run only if an extension installer drifted NumPy away from 2.5.3.

4. Verification Results

  • Verified that clean/repaired numpy==2.5.3 enables import torch, torchvision and torch.cuda.is_available() to succeed without errors.
  • Verified that consecutive startups skip redundant package uninstalls, cutting startup latency and eliminating the risk of partial-uninstall filesystem corruption.
  • Verified repository documentation and changelog alignment across md/CHANGELOG.md and README.md.

② Modified Files

File Change
modules/launch_utils.py Added pre-check NumPy validation/self-healing before PyTorch GPU test; guarded redundant package reinstalls
md/CHANGELOG.md Added v3.0.4 release notes and updated header
README.md Updated NumPy pin documentation reference to 2.5.3
Assets 2
Loading

v3.0.3 - NumPy 2.5.3 Upgrade — Complete Technical Explanation

v3.0.3 - NumPy 2.5.3 Upgrade — Complete Technical Explanation

Choose a tag to compare

View all tags
@ussoewwin ussoewwin released this 17 Sep 01:45
v3.0.3
d0e74f9

① Overview of the NumPy latest-version upgrade

1. Background (state before the change)

  • The NumPy in the venv was 2.4.6. The task was to update it to the latest release, 2.5.3.
  • The requirement files requirements_versions_py314*.txt were already updated to numpy==2.5.3 in the working tree (HEAD is numpy==2.4.6).
  • However, modules/launch_utils.py forcibly pinned NumPy to 2.4.6 at every startup, so even after raising the requirements the environment was reverted to 2.4.6 on each launch. That pin was the effective "setting", and it had to be raised together with the requirement files.

2. Latest versions (confirmed on PyPI)

Package Before After PyPI latest
numpy 2.4.6 2.5.3 2.5.3
numba 0.66.0 0.67.0 0.67.0
llvmlite 0.48.0 0.49.0 0.49.0

pip index versions confirmed the latest NumPy is 2.5.3 and the latest numba is 0.67.0.

3. What was changed

  1. venv packages
    • Installed numpy==2.5.3 (numpy-2.5.3-cp314-cp314-win_amd64.whl).
    • Upgraded numba to the latest: numba 0.66.0 → 0.67.0 (with llvmlite 0.48.0 → 0.49.0).
    • numba 0.67.0 requires numpy<2.6,>=1.22, so it coexists with numpy 2.5.3. When resolved together with numpy 2.5.3, this combination is selected automatically (pip install --dry-run shows Would install llvmlite-0.49.0 numba-0.67.0 numpy-2.5.3).
  2. Configuration side (in-code pin) aligned
    • Changed the NumPy force-pin in modules/launch_utils.py from 2.4.6 → 2.5.3 (three install sites plus related messages/comments).

4. Verification results

  • numpy 2.5.3 / numba 0.67.0 / llvmlite 0.49.0 all import successfully.
  • facexlib 0.3.0 (the package that depends on numba) imports successfully.
  • python -m pip check: the numba/NumPy conflict is resolved.
  • ast.parse syntax check on launch_utils.py passes.
  • No remaining 2.4.6 in the file (0 occurrences). No line-ending (EOL) changes.

Note: The following two pip check items are pre-existing and unrelated to the NumPy upgrade:

  • k-diffusion 0.0.16 → wandb not installed
  • pydantic-settings 2.15.0 → requires pydantic>=2.7.0 (currently 1.10.25)

② Files created / modified

Modified files

File Change
modules/launch_utils.py NumPy force-pin changed 2.4.6 → 2.5.3
requirements_versions_py314.txt numpy==2.4.6 → numpy==2.5.3 (updated in working tree)
requirements_versions_py314_windows.txt numpy==2.4.6 → numpy==2.5.3 (updated in working tree)

Created files

File Kind Content
md/NUMPY_253_PIN_UPDATE_2026-09-17.md Deliverable This document

Packages updated in the venv

  • venv/Lib/site-packages/numpy : 2.4.6 → 2.5.3
  • venv/Lib/site-packages/numba : 0.66.0 → 0.67.0
  • venv/Lib/site-packages/llvmlite : 0.48.0 → 0.49.0

③ Full text of created / modified code

3-1. modules/launch_utils.py (modified, before → after)

Before (HEAD)

    run_pip("install --upgrade packaging", "packaging")

    # Pin numpy==2.4.6 before -r (numba / facexlib compatible; not 2.5.x).
    # Install early so already-imported modules do not keep a stale numpy.
    run(f'"{python}" -m pip uninstall numpy -y', "uninstalling numpy", "Couldn't uninstall numpy", live=False)
    run(f'"{python}" -m pip install --no-cache-dir numpy==2.4.6', "Installing numpy 2.4.6", "Couldn't install numpy 2.4.6", live=False)
    print("[INFO] Installed numpy==2.4.6 from PyPI")

After

    run_pip("install --upgrade packaging", "packaging")

    # Pin numpy==2.5.3 before -r.
    # Install early so already-imported modules do not keep a stale numpy.
    run(f'"{python}" -m pip uninstall numpy -y', "uninstalling numpy", "Couldn't uninstall numpy", live=False)
    run(f'"{python}" -m pip install --no-cache-dir numpy==2.5.3', "Installing numpy 2.5.3", "Couldn't install numpy 2.5.3", live=False)
    print("[INFO] Installed numpy==2.5.3 from PyPI")

Before (HEAD)

    # scipy via PyPI to match numpy 2.4.6
    run(f'"{python}" -m pip uninstall scipy -y', "uninstalling scipy", "Couldn't uninstall scipy", live=False)
    run(f'"{python}" -m pip install --no-cache-dir scipy==1.16.1', "Installing scipy 1.16.1", "Couldn't install scipy 1.16.1", live=False)
    print("[INFO] Installed scipy 1.16.1 from PyPI")

After

    # scipy via PyPI to match numpy 2.5.3
    run(f'"{python}" -m pip uninstall scipy -y', "uninstalling scipy", "Couldn't uninstall scipy", live=False)
    run(f'"{python}" -m pip install --no-cache-dir scipy==1.16.1', "Installing scipy 1.16.1", "Couldn't install scipy 1.16.1", live=False)
    print("[INFO] Installed scipy 1.16.1 from PyPI")

Before (HEAD)

    if not args.skip_install:
        run_extensions_installers(settings_file=args.ui_settings_file)
        # Keep pin if an extension installer drifted numpy.
        run(f'"{python}" -m pip install --force-reinstall --no-deps --no-cache-dir numpy==2.4.6', "re-pin: numpy 2.4.6", "Couldn't install numpy 2.4.6", live=False)
        print("[INFO] Re-pinned numpy==2.4.6 after extensions")

After

    if not args.skip_install:
        run_extensions_installers(settings_file=args.ui_settings_file)
        # Keep pin if an extension installer drifted numpy.
        run(f'"{python}" -m pip install --force-reinstall --no-deps --no-cache-dir numpy==2.5.3', "re-pin: numpy 2.5.3", "Couldn't install numpy 2.5.3", live=False)
        print("[INFO] Re-pinned numpy==2.5.3 after extensions")

Reference: git diff (actual diff)

@@ -730,11 +730,11 @@ def prepare_environment():
 
     run_pip("install --upgrade packaging", "packaging")
 
-    # Pin numpy==2.4.6 before -r (numba / facexlib compatible; not 2.5.x).
+    # Pin numpy==2.5.3 before -r.
     # Install early so already-imported modules do not keep a stale numpy.
     run(f'"{python}" -m pip uninstall numpy -y', "uninstalling numpy", "Couldn't uninstall numpy", live=False)
-    run(f'"{python}" -m pip install --no-cache-dir numpy==2.4.6', "Installing numpy 2.4.6", "Couldn't install numpy 2.4.6", live=False)
-    print("[INFO] Installed numpy==2.4.6 from PyPI")
+    run(f'"{python}" -m pip install --no-cache-dir numpy==2.5.3', "Installing numpy 2.5.3", "Couldn't install numpy 2.5.3", live=False)
+    print("[INFO] Installed numpy==2.5.3 from PyPI")
 
     # Gradio from HF (METADATA version pins removed)
     run_pip(f'install "{gradio_package}"', "gradio")
@@ -746,7 +746,7 @@ def prepare_environment():
         run_pip(f"install -U --upgrade-strategy only-if-needed -r \"{requirements_file}\"", "requirements")
         startup_timer.record("install requirements")
     
-    # scipy via PyPI to match numpy 2.4.6
+    # scipy via PyPI to match numpy 2.5.3
     run(f'"{python}" -m pip uninstall scipy -y', "uninstalling scipy", "Couldn't uninstall scipy", live=False)
     run(f'"{python}" -m pip install --no-cache-dir scipy==1.16.1', "Installing scipy 1.16.1", "Couldn't install scipy 1.16.1", live=False)
     print("[INFO] Installed scipy 1.16.1 from PyPI")
@@ -806,8 +806,8 @@ def prepare_environment():
     if not args.skip_install:
         run_extensions_installers(settings_file=args.ui_settings_file)
         # Keep pin if an extension installer drifted numpy.
-        run(f'"{python}" -m pip install --force-reinstall --no-deps --no-cache-dir numpy==2.4.6', "re-pin: numpy 2.4.6", "Couldn't install numpy 2.4.6", live=False)
-        print("[INFO] Re-pinned numpy==2.4.6 after extensions")
+        run(f'"{python}" -m pip install --force-reinstall --no-deps --no-cache-dir numpy==2.5.3', "re-pin: numpy 2.5.3", "Couldn't install numpy 2.5.3", live=False)
+        print("[INFO] Re-pinned numpy==2.5.3 after extensions")
 
     if args.update_check:
         version_check(commit)

3-2. requirements_versions_py314.txt / requirements_versions_py314_windows.txt (modified)

Before (HEAD)

numpy==2.4.6

After

numpy==2.5.3

3-3. Replacement script (local, not committed)

The pin strings in modules/launch_utils.py were replaced with a short local Python script (byte-safe: read as bytes, decode UTF-8, replace, write back, so line endings are preserved). It is kept outside the repository and is not part of the commit.

P = "modules/launch_utils.py"

with open(P, "rb") as f:
    raw = f.read()
s = raw.decode("utf-8")

reps = [
    # 1) comment line: drop the now-false "not 2.5.x" rationale, pin 2.5.3
    (
        "    # Pin numpy==2.4.6 before -r (numba / facexlib compatible; not 2.5.x).",
        "    # Pin numpy==2.5.3 before -r.",
    ),
    # 2) exact version specifiers
    ("numpy==2.4.6", "numpy==2.5.3"),
    # 3) human-readable messages / comments that spell the version with a space
    ("numpy 2.4.6", "numpy 2.5.3"),
]

for old, new in reps:
    print(old, s.count(old))
    s = s.replace(old, new)

with open(P, "wb") as f:
    f.write(s.encode("utf-8"))

print("remaining 2.4.6:", [l for l in s.splitlines() if "2.4.6" in l])

3-4. Commands executed

# 1) Inspect current state
& 'venv\Scripts\python.exe' -c "import sys, numpy; print(sys.version); print(numpy.__version__)"

# 2) Upgrade numpy to 2.5.3
& 'venv\Scripts\python.exe' -m pip install "numpy==2.5.3"

# 3) Upgrade numba to latest (resolved together with numpy)
& 'venv\Scripts\python.exe' -m pip install -U numba

# 4) Verify
& 'venv\Scripts\python.exe' -c "import numpy, numba, llvmlite; print(numpy.__version__, numba.__version__, llvmlite.__version__)"
& 'venv\Scripts\python.exe' -m pip check

④ What it means

1. The two lay...

Read more
Loading

v3.0.2 - Direct Flash-Attention 2 Auto-Detection, Priority & Forge Kernel Proof Logging

v3.0.2 - Direct Flash-Attention 2 Auto-Detection, Priority & Forge Kernel Proof Logging

Choose a tag to compare

View all tags
@ussoewwin ussoewwin released this 04 Sep 07:53
v3.0.2
1400371

v3.0.2 - Direct Flash-Attention 2 Auto-Detection, Priority & Forge Kernel Proof Logging

Summary:
v3.0.2 restores direct Flash-Attention 2 auto-detection, priority assignment, and execution visibility in A1111, adhering strictly to the Forge-Nunchaku (backend/attention_backend_info.py) kernel proof principles and A1111's direct-load architectural design (md/FA2_direct_load_design.md).

Highlights

Component / Area Before (v3.0.1) After (v3.0.2)
FA Direct Detection Only attempted if --flash-attention CLI flag passed Automatically detected and imported if flash_attn exists in environment
FA Optimizer Priority priority = 50 (subordinate to Doggettx 90 & SDP 80/70) priority = 110 (highest priority, above xformers 100, Doggettx 90, SDP 80/70)
xformers Coupling is_available() restricted by not shared.xformers_available Decoupled: Direct FA-2 available regardless of whether xformers is installed
Automatic Selection Fell back to Doggettx / xformers unexpectedly Selects Direct Flash-Attention by default unless explicit CLI flag given
Attention Kernel Proof Logging Hardcoded unconditional print in process_images_inner Forge-standard proof log emitted only after real CUDA kernel execution
Failure Observability Silent fallback / obscured Explicit notice [A1111] FA-2 kernel NOT used — ... fallback pytorch SDPA

1. Direct Flash-Attention 2 Auto-Detection (modules/sd_hijack_optimizations.py)

  • Background: Upstream direct-load design allows Flash-Attention 2 (flash_attn_func) to execute directly without xformers. Previously, import was gated behind if shared.cmd_opts.flash_attention:, leaving FLASH_ATTN_AVAILABLE = False during standard launches without CLI arguments.
  • Change: Directly attempts import flash_attn and resolves version/type (FA-2 or FA-3) on startup using module inspection helpers (_module_version, _module_file, _callable_file).
  • Startup Console Output: 
    [A1111] Flash-Attention direct import: FA-2 version <ver>

2. Optimizer Priority & Automatic Selection (modules/sd_hijack_optimizations.py & modules/sd_hijack.py)

  • Background: In previous versions, SdOptimizationFlashAttn.priority was set to 50 and is_available() required not shared.xformers_available. When xformers was installed or when running with default settings, Doggettx (priority 90) or xformers took precedence, logging Applying attention optimization: Doggettx... done. during VAE/checkpoint switches.
  • Change:
    • SdOptimizationFlashAttn.priority raised to 110.
    • Removed not shared.xformers_available restriction.
    • In modules/sd_hijack.py, apply_optimizations() honors explicit CLI arguments (e.g. --xformers, --opt-sdp-attention) if provided; otherwise, it selects the highest priority available optimizer (SdOptimizationFlashAttn at 110).
  • Model Load Console Output: 
    Applying attention optimization: flash-attention... done.

3. Forge-Standard Attention Kernel Proof Logging

  • Background: Follows Forge-Nunchaku (backend/attention_backend_info.py) kernel proof rules:
    1. Never claim a kernel ran before it executes.
    2. Proof logs are emitted strictly after the real CUDA/Python kernel returns.
    3. Never print called for a PyTorch fallback.
    4. Output once per generation job.
  • Implementation:
    • In flash_attention_forward() and flash_attn_attnblock_forward(), inspects the callable module and disk file location after flash_attn_func(...) completes.
    • Outputs proof line: 
      [A1111] FA-2 kernel ran: flash_attn.flash_attn_interface.flash_attn_func ver=<version> file=<path>
    • If kernel execution encounters an error, logs honest fallback notice before switching to PyTorch SDPA: 
      [A1111] FA-2 kernel NOT used — flash_attn_func failed (<exception>); fallback pytorch SDPA

4. Removal of Premature / False Logs (modules/processing.py)

  • Background: modules/processing.py contained an unconditional print statement inside process_images_inner(): 
    print("[A1111] xformers.memory_efficient_attention called (FA-3→FA-2→Cutlass priority order)")
    This line executed on every generation job regardless of whether xformers, Doggettx, or flash-attention was active.
  • Change: Removed the premature hardcoded log statement. Real execution proof is now delegated strictly to kernel return hooks.

Verification & Compatibility

  • Python Version: CPython 3.14 (aligned with v3.0.0+ platform renewal).
  • PyTorch / CUDA Stack: PyTorch 2.13.0+cu132 with Flash-Attention 2 2.8.4+cu132torch2.13.0.
  • Fallback Guarantee: If flash_attn is unavailable or fails at runtime, execution cleanly degrades to SDPA then sub_quad_attention, maintaining zero generation aborts.
Loading

v3.0.1 - SDXL ConvRot INT8 / NVFP4 Checkpoint Support

v3.0.1 - SDXL ConvRot INT8 / NVFP4 Checkpoint Support

Choose a tag to compare

View all tags
@ussoewwin ussoewwin released this 13 Aug 13:34
v3.0.1
2b4b237

This release adds support for loading SDXL ConvRot INT8 and SDXL ConvRot NVFP4 checkpoints (ComfyUI-native comfy_quant format) directly in A1111. Quantized weights are offline-dequantized and Hadamard-unrotated at load time, then flow through the standard float16 pipeline — with complete isolation from all other model formats.

✨ What's New

1. SDXL ConvRot INT8 Checkpoint Loading

  • New module modules/sdxl_int8_convrot.py + a fully isolated hook in modules/sd_models.py (96a80ed7)
  • Detects checkpoints via .comfy_quant keys (format int8_tensorwise, optional "convrot":true)
  • Supports all three INT8 scale layouts:
    • Plain INT8 — scalar weight_scale (tensorwise)
    • ConvRot Linear — row-wise weight_scale [out, 1], unrotated via W = W_rot @ H
    • ConvRot Conv2d — channel-wise weight_scale [out, 1, 1, 1], unrotated along in_channels
  • Sidecar keys (.weight_scale, .comfy_quant, .input_scale, .weight_scale_2, .pre_quant_scale) are fully cleaned up after dequantization

2. SDXL ConvRot NVFP4 Checkpoint Loading

  • NVFP4 (E2M1 packed) Linear layers dequantized via comfy_kitchen.dequantize_nvfp4 (27891d83)
  • Handles the mixed-pack layout: Linear = NVFP4, Conv2d = INT8 tensorwise, CLIP/VAE = float
  • Padded → logical shape crop via orig_shape from comfy_quant metadata
  • ConvRot Hadamard unrotate applied for "convrot":true layers
  • Respects hi_first nibble ordering (default true)

3. Unified Dequantize Pipeline

  • dequantize_state_dict() runs NVFP4 pass first, then INT8 pass — both in-place, before checkpoint caching and load_state_dict
  • Cache stores post-dequant float weights, so reloads skip re-processing
  • Console log reports per-format layer counts (int8_plain, int8_convrot_linear, int8_convrot_conv2d, nvfp4_plain, nvfp4_convrot, etc.)

4. Docs

  • Full technical documentation: docs/SDXL_ConvRot_INT8_NVFP4_Technical_Documentation.md (07fa0b3f)

🔒 Isolation Guarantee (Design Requirement)

  • Strict no-op for any checkpoint without .comfy_quant keys
  • Format-specific branching: only int8_tensorwise → INT8 path, only nvfp4 → NVFP4 path
  • Zero monkey-patches, zero custom layer injection, zero forward-path changes
  • After loading, the model is an ordinary float16 torch.nn.Module — LoRA, ControlNet, VAE, embeddings all work exactly as usual
  • SD1.5 / SD2 / SDXL (non-quantized) / fp16 / fp8 / bnb checkpoints are completely unaffected

🐛 Bug Fixes

  • Stats key collision between NVFP4 and INT8 passes in the combined stats dict — NVFP4 counter renamed to nvfp4_skipped (181e3783)

🧪 Testing

  • INT8 unit tests (6): detection, plain dequantize (max diff < 0.018), ConvRot Linear (< 0.040), ConvRot Conv2d (< 0.031), float-weight skip, no-op on plain state_dict
  • NVFP4 unit tests (4): plain dequantize (< 0.68), ConvRot dequantize (< 0.42), mixed-pack NVFP4+INT8, no-op on plain state_dict
  • Verified against the A1111 load pipeline: find_checkpoint_config, set_model_type, instantiate_from_config, LoadStateDictOnMeta, sd_disable_initialization (int8 Parameter error resolved), checkpoint cache, reload_model_weights

📁 Files Changed

File Status Description
modules/sdxl_int8_convrot.py New Hadamard build/unrotate, INT8 + NVFP4 dequantize, combined entry point
modules/sd_models.py Modified Isolated dequantize hook in load_model_weights()
docs/SDXL_ConvRot_INT8_NVFP4_Technical_Documentation.md New Full technical reference
md/CHANGELOG.md Modified v3.0.1 release entry

Commits

96a80ed7 feat: add SDXL INT8 ConvRot checkpoint support
27891d83 feat: add SDXL NVFP4 ConvRot checkpoint support
181e3783 fix: stats key collision between NVFP4 and INT8 skipped counters
07fa0b3f docs: add SDXL ConvRot INT8/NVFP4 technical documentation
4b4e1914 chore: add v3.0.1 changelog entry (SDXL ConvRot INT8/NVFP4 support)

⚠️ Notes

  • VRAM: weights are dequantized to float16 at load time, so VRAM usage equals a standard float16 model (no runtime quantization savings — the INT8/NVFP4 file-size benefit is preserved on disk)
  • Dependency: NVFP4 requires comfy_kitchen in the A1111 venv (already present in this fork)
  • Inference speed: identical to standard float16 models
  • Target: Python 3.14 fork (A1111-for-Python3.14)
Loading

MAJOR UPDATE: v3.0.0 - Python 3.14 Full Renewal

MAJOR UPDATE: v3.0.0 - Python 3.14 Full Renewal

Choose a tag to compare

View all tags
@ussoewwin ussoewwin released this 09 Aug 16:01
v3.0.0
fb86672

🎉 MAJOR UPDATE: v3.0.0 - Python 3.14 Full Renewal!

First release of ussoewwin/A1111-for-Python3.14. The fork is now Python 3.14–only, with a renewed default CUDA stack (torch==2.13.0+cu132 / FA2 2.8.4) and critical CPython 3.14 runtime fixes.

Overview

v3.0.0 is the first release of this fork as ussoewwin/A1111-for-Python3.14.

Through v2.3.5, the published surface was ussoewwin/A1111-for-Python3.12. From v3.0.0, the fork is a Python 3.14–only renewal: install docs, launch scripts, dependency pins, Flash-Attention 2, and runtime gates all assume CPython 3.14. Other Python versions are not supported.

Tag commit: d9433e42
Repository: https://github.com/ussoewwin/A1111-for-Python3.14

Highlights

Area v3.0.0
Python 3.14 only (check_python_version / README / webui.sh)
Default CUDA torch (Win/Linux) torch==2.13.0+cu132 + torchvision==0.28.0+cu132
Torchaudio Not in the default first-install command (cu132 index has no matching torchaudio; audio paths lazy-import)
Flash-Attention 2 2.8.4 aligned to torch 2.13.0+cu132
CUDA toolkit (Linux FA2 source build) 13.2 + nvcc
NumPy / SciPy numpy==2.5.1 and scipy==1.16.1 from PyPI (all platforms)
Requirements files Renamed to requirements_versions_py314*.txt
Critical runtime fix CPython 3.14 shared.sd_model LOAD_ATTR / None shadow

1. Python 3.14–only platform

What changed

  • Install and platform docs state: This repository supports Python 3.14 only.
  • modules/launch_utils.py rejects other majors/minors unless --skip-python-version-check is used (not recommended).
  • webui.sh / webui-macos-env.sh defaults aligned to the 3.14 / torch 2.13 stack.
  • Dependency requirement files renamed from *_py312* to *_py314* / *_py314_windows*.
  • Distutils stub install retained for 3.14 (stdlib distutils removed upstream).

Migration from A1111-for-Python3.12

  1. Install Python 3.14.
  2. Recreate venv (do not reuse a 3.12 venv).
  3. Follow README Windows / Linux / macOS sections for torch + FA2.
  4. Expect extension compatibility gaps; not all extensions are verified on 3.14.

2. Default first-install CUDA stack (Windows / Linux)

Default environment variables / command (overridable):

TORCH_INDEX_URL = https://download.pytorch.org/whl/cu132
TORCH_COMMAND   = pip install torch==2.13.0+cu132 torchvision==0.28.0+cu132 --index-url <TORCH_INDEX_URL>

Design goals

  • Same pin on Windows and Linux so Linux FA2 source builds against the same torch ABI family as the Windows FA2 wheel.
  • Match Flash-Attention 2 Windows artifact: 2.8.4+cu132torch2.13.0 (cp314).
  • Omit torchaudio from the default install: the cu132 index does not ship a matching torchaudio for this pin. ComfyUI-master paths that need torchaudio use lazy imports so missing torchaudio does not break unrelated model loads.

Override

Set TORCH_COMMAND / TORCH_INDEX_URL before launch if you need another CUDA / CPU build.

3. Flash-Attention 2 (2.8.4)

Platform Method
Windows Hugging Face prebuilt wheel flash_attn-2.8.4+cu132torch2.13.0 (cp314), via FLASH_ATTN_PACKAGE / launch_utils
Linux PyPI flash-attn==2.8.4 source build with --no-build-isolation against installed torch==2.13.0+cu132; needs CUDA toolkit 13.2 and nvcc on PATH (often ~30 minutes)
macOS FA2 skipped (MPS limitation)

README Platform Support / Default Package Versions / Linux install steps were rewritten to this stack (not older FA2 2.9.x / cu130 pins).

4. CPython 3.14 shared.sd_model LOAD_ATTR / None shadow (critical)

Symptom class

On CPython 3.14, after the first successful property access, specializing LOAD_ATTR could return None for shared.sd_model / p.sd_model even while model_data still held a live model. ControlNet, MultiDiffusion, Tiled VAE, and checkpoint switch paths then saw sd_model is None and failed or behaved inconsistently.

Root cause (fork analysis)

If both:

  1. Shared.sd_model as a data descriptor (property), and
  2. modules.shared.__dict__['sd_model'] = None

exist, 3.14 attribute load specialization can prefer the dict slot and return the shadowed None.

Countermeasure (high level)

  • Do not bind sd_model = None into modules.shared module dict; keep annotation / Shared property only (modules/shared.py, modules/shared_items.py).
  • Strengthen load / keep-old / Processing paths so callers read the live model from model_data / guarded accessors (modules/sd_models.py, modules/processing.py).
  • Tiled VAE / related guards updated so they do not assume a stale None module attribute (extensions-builtin/.../tilevae.py and related).

Commit: 88646c18 — fix: CPython 3.14 shared.sd_model LOAD_ATTR None shadow

5. Checkpoint / VAE switch guards and LoRA safety

Part of the 3.14 defaults commit (8588e0ca):

  • Guards around model / VAE switch so callers do not hit NoneType during checkpoint transitions.
  • LoRA network path adjustments for safer behavior when the active model is mid-switch.
  • Related touch-ups in textual inversion / sd_vae helpers.

6. NumPy / SciPy (numpy==2.5.1)

  • Startup installs numpy==2.5.1 from PyPI on all platforms (modules/launch_utils.py), then pins again after extension installers so drift cannot stick.
  • Requirements: requirements_versions_py314.txt / requirements_versions_py314_windows.txt pin numpy==2.5.1.
  • SciPy is scipy==1.16.1 from PyPI, aligned to that NumPy pin (no Windows-only HF scipy wheel forced onto Linux).
  • Interim whl/numpy-1.26.4-cp314-… work (commit 15d8a90b) was never the v3.0.0 install path; runtime installs NumPy from PyPI only. That interim wheel is removed from the tree so docs cannot point at a local 1.26.4 artifact as current.

7. Vendored ComfyUI-master sync

ComfyUI-master/ (used for RES4LYF / comfy.* runtime) was synced with upstream ComfyUI updates in 8588e0ca, including newer models / nodes / API surfaces.

Follow-up in 3b873426: lazy torchaudio imports in:

  • ComfyUI-master/comfy/text_encoders/gemma4.py
  • ComfyUI-master/comfy/ldm/lightricks/vae/audio_vae.py

so optional audio dependencies are not required for the default A1111 torch install.

8. Documentation

  • README: Python 3.14-only, Platform Support table, Default Package Versions, Windows / Linux / macOS install aligned to torch 2.13.0+cu132 and FA2 2.8.4.
  • Compatibility / Linux–macOS notes under md/ updated for the same stack.
  • md/CHANGELOG.md: header updated through v3.0.0; notes the transition from A1111-for-Python3.12 to A1111-for-Python3.14.

9. Commits included in this tag (Python 3.14 renewal line)

8588e0ca  feat: Python 3.14 defaults, model/VAE switch guards, sync ComfyUI-master
15d8a90b  feat: replace tracked numpy wheel cp312 with cp314
88646c18  fix: CPython 3.14 shared.sd_model LOAD_ATTR None shadow
3b873426  feat: default torch 2.13.0+cu132 and align Linux README/FA2
d9433e42  docs: add CHANGELOG v3.0.0 Python 3.14 renewal overview

Earlier v2.3.x work (RES4LYF, Forge tiled VAE, ControlNet bf16, LoRA logging, etc.) remains in history from the 3.12-era lineage; this release focuses on the 3.14 renewal delta.

10. Suggested verify after upgrade

  1. Fresh Python 3.14 venv; confirm startup does not reject the interpreter.
  2. Confirm installed torch prints 2.13.0+cu132 (or your override) and matching torchvision.
  3. Confirm numpy==2.5.1 (and scipy==1.16.1) from PyPI — not a local whl/numpy-1.26.4 install.
  4. Windows: FA2 wheel installs; Linux: FA2 source build completes with toolkit 13.2.
  5. Load SD1.5 / SDXL, run txt2img with ControlNet and/or MultiDiffusion if you use them — confirm no intermittent sd_model is None.
  6. Switch checkpoints / VAE mid-session without LoRA / VAE NoneType crashes.
  7. Optional: RES4LYF hybrid *4h4s still completes (shim from prior releases).

Links

Loading

v2.3.5 - RES4LYF Hybrid Sampler — `data_prev_` IndexError Fix

v2.3.5 - RES4LYF Hybrid Sampler — `data_prev_` IndexError Fix

Choose a tag to compare

View all tags
@ussoewwin ussoewwin released this 11 Jul 03:41
v2.3.5
dcce1fb

1. Error Content

1-1. Console / progress

Sampling starts and advances for several steps, then aborts mid-run (example: 5/16):

(RES4LYF) rk_type: lawson45-gen-mod_4h4s
  0%|                                                                                           | 0/16 [00:00<?, ?it/s]
[A1111] FA-2 (Flash-Attention 2.9.1) called directly
 31%|█████████████████████████▉                                                         | 5/16 [00:17<00:34,  3.12s/it]
*** Error completing request

1-2. Exception

IndexError: index 4 is out of bounds for dimension 0 with size 4

1-3. Stack (abbreviated)

File "modules/call_queue.py", line 74, in f
File "modules/txt2img.py", line 109, in txt2img
File "modules/processing.py", line 848 / 1005, in process_images / process_images_inner
File "extensions-builtin/sd-webui-controlnet/scripts/hook.py", line 470, in process_sample
File "modules/sd_samplers_kdiffusion.py", line 230, in sample
File "modules/a1111_res4lyf_samplers.py", line 274, in wrapped_func
File "modules/a1111_res4lyf_samplers.py", line 197, in sample_ode_fn
File "modules/RES4LYF/beta/rk_sampler_beta.py", line 2029, in sample_rk_beta
    data_prev_[recycled_stages - ms] = data_prev_[recycled_stages - ms - 1]
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
IndexError: index 4 is out of bounds for dimension 0 with size 4

1-4. Affected samplers (this bug class)

Any RES4LYF hybrid RK type whose name ends in NhMs with N >= 4, so that:

hybrid_stages = int(rk_type[-4])  # e.g. lawson45-gen-mod_4h4s → 4

and recycled_stages becomes 4 while data_prev_ still has only 4 slots (valid indices 0..3).

Confirmed failing example: lawson45-gen-mod_4h4s / lawson45-gen-mod_4h4s_ode.

Early steps may still run because hybrid methods often fall back to res_* for the first few steps (rk_coefficients_beta.py hybrid gate). The crash appears once the real hybrid stage is active and the history-rotation loop runs with recycled_stages == 4.

2. Essential Root Cause

2-1. Hardcoded buffer size 4

In modules/RES4LYF/beta/rk_sampler_beta.py, multistep / hybrid history buffers are allocated as:

data_prev_ = torch.zeros(4, *x.shape, dtype=default_dtype, device=work_device)
# comment in source: "multistep max is 4m... so 4 needed"

Same pattern for sync-guide buffers:

data_prev_x_ = torch.zeros(4, *x.shape, ...)
data_prev_y_ = torch.zeros(4, *x.shape, ...)

(lines 744, 746, 1134, 1135).

2-2. hybrid_stages can be 4

In modules/RES4LYF/beta/rk_coefficients_beta.py:

if rk_type[-3] == "h" and rk_type[-1] == "s":  # hybrid method
    ...
    hybrid_stages = int(rk_type[-4])

For lawson45-gen-mod_4h4s, the digit before h is 4 → hybrid_stages = 4.

2-3. recycled_stages takes the max

In rk_sampler_beta.py (around line 725):

recycled_stages = max(rk_swap_stages, RK.multistep_stages, RK.hybrid_stages, data_prev_len)

When RK.hybrid_stages == 4, recycled_stages becomes 4 (after the first init loop that temporarily sets recycled_stages = len(data_prev_)-1 == 3).

2-4. Rotation loop writes index 4

At the end of each step (lines 2027–2029):

data_prev_[0] = data_[0]
for ms in range(recycled_stages):
    data_prev_[recycled_stages - ms] = data_prev_[recycled_stages - ms - 1]

With recycled_stages = 4 and ms = 0:

data_prev_[4] = data_prev_[3]

But data_prev_.shape[0] == 4 → valid indices are only 0, 1, 2, 3 → IndexError.

2-5. Why A1111 cannot “just edit RES4LYF”

modules/RES4LYF/ is third-party / AGPL-attributed code kept unmodified in this fork’s policy for this fix. The correct place for the workaround is the existing A1111 bridge: modules/a1111_res4lyf_shim.py, which already wraps every RES4LYF sampler call via res4lyf_shim_context.

2-6. Causal chain (summary)

UI sampler: lawson45-gen-mod_4h4s(_ode)
    → hybrid_stages = 4
    → recycled_stages = 4
    → data_prev_ allocated as torch.zeros(4, ...)
    → rotation writes data_prev_[4]
    → IndexError

3. Countermeasure

① Overview

Do not change RES4LYF sources. Inside res4lyf_shim_context (already entered for every RES4LYF sample via a1111_res4lyf_samplers.wrapped_func):

  1. Resolve the live module object RES4LYF.beta.rk_sampler_beta (with fallbacks).
  2. Temporarily replace that module’s torch attribute with a thin proxy.
  3. The proxy forwards all attributes to real torch, but intercepts zeros: if the first positional argument is the integer 4, allocate with first dim 8 instead.
  4. On context exit (finally), restore the original torch on the module.

Effect: data_prev_ / data_prev_x_ / data_prev_y_ become size 8 on dim 0 → index 4 is valid. Other torch.zeros calls with a different leading dim are unchanged. Global torch for the rest of A1111 is not patched.

Why 8: hybrid_stages max in current name list is 4 → need at least 5 slots (0..4). 8 gives headroom without touching RES4LYF.

Module name note: An early draft looked up beta.rk_sampler_beta only; that name does not resolve at A1111 runtime. The live name is RES4LYF.beta.rk_sampler_beta. Without the correct name, the proxy never installed and the IndexError remained.

② Modified file names

Path Role
modules/a1111_res4lyf_shim.py Runtime fix (patch inside res4lyf_shim_context)
md/A1111_RES4LYF_INTEGRATION.md Integration doc synced to include the same patch block

Not modified: anything under modules/RES4LYF/.

③ Full text of the added / modified code

Import addition (top of a1111_res4lyf_shim.py)

import sys

(alongside existing inspect, logging, contextmanager, torch).

Patch block inside res4lyf_shim_context (after diffusion_model alias, before yield)

    # --- 3. patch RES4LYF rk_sampler_beta torch.zeros for data_prev_* buffers ---
    #
    # RES4LYF allocates ``data_prev_`` / ``data_prev_x_`` / ``data_prev_y_``
    # as ``torch.zeros(4, *x.shape, ...)`` (rk_sampler_beta.py lines 744, 746,
    # 1134, 1135) — hardcoded size 4. For hybrid samplers like
    # ``lawson45-gen-mod_4h4s`` where ``hybrid_stages = 4``, ``recycled_stages``
    # becomes 4 (rk_sampler_beta.py line 725), and the rotation loop at line
    # 2028-2029 writes ``data_prev_[4]`` → IndexError (size 4, valid 0..3).
    #
    # We cannot edit ``modules/RES4LYF/``. Instead, we replace ``torch.zeros``
    # **only in the ``rk_sampler_beta`` module's globals** (not the global
    # ``torch`` package) with a wrapper that grows any size-4 first-dim
    # allocation to 8 — enough for all current hybrid_stages values
    # (max 4 → need 5 slots, 8 gives headroom). Other torch.zeros calls
    # (different leading dim) pass through unchanged.
    rk_sampler_beta_mod = None
    try:
        import importlib
        # RES4LYF is imported as a subpackage of ``modules``; the fully
        # qualified name is ``RES4LYF.beta.rk_sampler_beta``. The bare
        # ``beta.rk_sampler_beta`` form only resolves when CWD is inside
        # the RES4LYF package, which is not the case at runtime.
        for _modname in (
            "RES4LYF.beta.rk_sampler_beta",
            "modules.RES4LYF.beta.rk_sampler_beta",
            "beta.rk_sampler_beta",
        ):
            _m = sys.modules.get(_modname)
            if _m is not None:
                rk_sampler_beta_mod = _m
                break
        if rk_sampler_beta_mod is None:
            for _modname in (
                "RES4LYF.beta.rk_sampler_beta",
                "modules.RES4LYF.beta.rk_sampler_beta",
                "beta.rk_sampler_beta",
            ):
                try:
                    rk_sampler_beta_mod = importlib.import_module(_modname)
                    break
                except Exception:
                    continue
    except Exception:
        rk_sampler_beta_mod = None

    zeros_patched = False
    _rk_torch = None
    if rk_sampler_beta_mod is not None:
        # Inject a module-level helper that shadows torch.zeros only inside
        # rk_sampler_beta's global scope. We do this by replacing the
        # ``torch`` attribute the module sees with a lightweight namespace
        # that proxies all torch.* calls but intercepts ``zeros``.
        try:
            _rk_torch = rk_sampler_beta_mod.torch
            _orig_torch_zeros = _rk_torch.zeros

            def _patched_zeros(*args, **kwargs):
                if args and isinstance(args[0], int) and args[0] == 4:
                    args = (8,) + args[1:]
                return _orig_torch_zeros(*args, **kwargs)

            # Build a proxy that forwards every attribute to real torch except
            # ``zeros`` which goes through our wrapper.
            class _TorchProxy:
                def __getattr__(self, name):
                    return getattr(_rk_torch, name)

                def zeros(self, *args, **kwargs):
                    return _patched_zeros(*args, **kwargs)

            rk_sampler_beta_mod.torch = _TorchProxy()
            zeros_patched = True
        except Exception:
            logger.debug("[RES4LYF shim] Failed to patch rk_sampler_beta.torch.zeros", exc_info=True)

    try:
        yield
    finally:
        # --- restore rk_sampler_beta.torch ---
        if zeros_patched and rk_sampler_beta_mod is not None:
            try:
                rk_sampler_beta_mod.torch = _rk_torch
            except Exception:
                logger.debug("[RES4LYF shim] restore rk_sampler_beta.torch fail...
Read more
Loading

v2.3.4 — LoRA load logging and SDXL mapping fixes\

v2.3.4 — LoRA load logging and SDXL mapping fixes\

Choose a tag to compare

View all tags
@ussoewwin ussoewwin released this 01 Jul 04:31
v2.3.4
97de926

v2.3.4 — LoRA load logging and SDXL mapping fixes

Added

  • LoRA load logging — when the active LoRA stack changes, extensions-builtin/Lora/networks.py prints per-file UNet/CLIP load lines (loaded key count, multiplier, skipped keys) to the console; duplicate prompts are suppressed via a fingerprint of loaded networks (3798a083).

Fixed

  • SDXL LoRA layer mapping — SDXL upsampler conv keys map to output_blocks_*_2_conv (not SD1.x type=1 on up_blocks_0); SDXL CLIP/OpenCLIP alias keys and Forge-style diffusers lookup paths so more LoRA keys resolve (3798a083).

Summary

Console LoRA load diagnostics and SDXL-focused network layer mapping fixes.

Loading

\ v2.3.3 — RES4LYF samplers and schedulers\

\ v2.3.3 — RES4LYF samplers and schedulers\

Choose a tag to compare

View all tags
@ussoewwin ussoewwin released this 01 Jul 04:06
v2.3.3
b577abb

v2.3.3 — RES4LYF samplers and schedulers

Added

  • RES4LYF samplers and schedulers — ComfyUI RES4LYF integration for A1111 via modules/a1111_res4lyf_samplers.py and modules/a1111_res4lyf_shim.py. Registers RES4LYF-derived sampling methods (including dynamic RK variants) plus schedulers beta57 and bong_tangent in the native txt2img/img2img UI. Vendored ComfyUI-master/ on sys.path for comfy.* runtime imports; modules/RES4LYF/ is left unmodified.
  • Startup hook in modules/initialize.py — calls RES4LYF registration immediately after sd_samplers.set_samplers().

Fixed

  • RES4LYF / A1111 sampler name collision — when a RES4LYF sampler name matches a built-in A1111 sampler (e.g. dpmpp_2m, euler), the RES4LYF entry is skipped so standard samplers keep working (a194ea56).

Summary

Native A1111 UI support for RES4LYF samplers and schedulers (Forge-style glue + execution shim).

Technical details

Full technical write-up: A1111 RES4LYF Integration

Loading

v2.3.2 - Forge Tiled VAE Encode NaN / narrow Crash Fix

v2.3.2 - Forge Tiled VAE Encode NaN / narrow Crash Fix

Choose a tag to compare

View all tags
@ussoewwin ussoewwin released this 28 Jun 01:07
v2.3.2
b4263ad

Typical failing pipeline: img2img upscale, MultiDiffusion + ControlNet tile + Noise Inversion, Forge Tiled VAE enabled, ~16 GB VRAM. Example input: 1086×954 px (latent 136×120 after encode).

1. What Failed (Error Symptoms)

1-1. Primary failure: NaNs in UNet

After Forge tiled VAE encode, sampling aborts at step 0:

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

or (more often, after device fix):

NansException: A tensor with NaNs was produced in Unet

With diagnostic logging enabled ([MD-DIAG] in _forge_3pass_tiled), the NaNs appear inside the encode passes, not in the final average:

[MD-DIAG] 3pass[0] tile=(192,192) ov=24 part shape=(1,4,120,136) nan=0 inf=0 absmax=...
[MD-DIAG] 3pass[1] tile=(384,128) ov=24 part shape=(1,4,120,136) nan=624 inf=0 absmax=...
[MD-DIAG] 3pass[2] tile=(128,384) ov=24 part shape=(1,4,120,136) nan=0 inf=0 absmax=...
[MD-DIAG] init_latent ... nan=True ...

624 NaNs on pass [1] (wide tile 384×128 with overlap 24) was the smoking gun for image H=966 (and similarly for H=954 with different pass geometry).

1-2. Secondary failure (superseded intermediate fix)

After commit 019e419d (replicate-pad encoder input to multiple of 8), some heights that already satisfied floor(H/8) == round(H/8) overshot the latent buffer and crashed during accumulation:

RuntimeError: The size of tensor a (121) must match the size of tensor b (120) at non-singleton dimension 3

or PyTorch narrow errors when slicing the output tensor. Example: H=954 → latent 119×120 expected, but padded encode produced 120×120 latent rows.

2. Root Cause

Forge tiled encode runs three tiled_scale_multidim(..., downscale=False) passes over the same pixel image, then averages the three latent results (acc / 3.0). Each pass uses different tile shapes from _encode_passes:

((base, base), overlap),
((base * 2, max(base // 2, 128)), overlap),   # pass[1]: wide
((max(base // 2, 128), base * 2), overlap),   # pass[2]: tall

Inside tiled_scale_multidim, each tile is VAE-encoded and accumulated into a fixed output buffer out, then normalized:

out[b, :, ...] += mask * ps
out_div[b, :, ...] += mask
# ...
output[b] = out / out_div

2-1. floor(N/8) vs round(N/8) mismatch

SD1.5 / SDXL VAE encoders use stacked stride-2 convolutions with asymmetric padding (pad=(0,1,0,1) on some layers). For pixel height H:

Quantity Formula H=966 H=954
VAE latent rows (actual) floor(H/8) 120 119
tiled_scale buffer rows round(H/8) 121 119

The output tensor is allocated with round:

output = torch.empty(
    [samples.shape[0], out_channels] + mult_list_upscale(samples.shape[2:]),
    ...
)
# mult_list_upscale uses round(get_pos(d, pos)) with upscale_amount=1/8

Each tile's VAE output ps has size based on floor of that tile's pixel extent. When a last-axis tile does not fill the remaining buffer slots, some latent indices never receive out_div > 0. After division:

0 / 0  →  NaN

2-2. Why pass [1] produced 624 NaNs (H=966)

For H=966, pass [1] uses tile (384, 128). Because 966 < 384 on the Y axis, the tile loop runs once at y=0 with length=966 — a single tile covers the full image height.

  • Buffer latent height: round(966/8) = 121
  • VAE output height: floor(966/8) = 120
  • Row 0 of the buffer slot is never written → 121 NaNs per channel × 4 channels × ~156 cols ≈ 624 (matches log).

Pass [0] and [2] use smaller tiles on at least one axis, so multiple tile positions overlap and mask-weighted blending covers the buffer. Pass [1] is the problematic single-tile full-axis case.

2-3. Why end-align (24aefab9) was insufficient

Commit 24aefab9 added end-align: shift upscaled[d] so the last tile's output sits at the bottom/right of the buffer:

for d in range(dims):
    if last_axis[d] and upscaled[d] + mask.shape[d + 2] < out.shape[d + 2]:
        upscaled[d] = max(0, out.shape[d + 2] - mask.shape[d + 2])

This fixes multi-tile last rows (e.g. width 2325 → latent 291, last tile covers 286–289, index 290 empty). It fails when one tile spans the entire axis: shifting the 120-row output to index 1..120 leaves index 0 uncovered → same 0/0 NaN.

2-4. Why encoder-input padding (019e419d) was wrong place

Padding the encoder input to the next multiple of 8 forces floor(H'/8) == round(H'/8) for padded H'. When already floor(H/8) == round(H/8) (e.g. H=954 → 119), padding adds 8 px → H'=962 → latent 120×120, but the buffer remains 119×120 → narrow / size mismatch crash.

The fix must pad the VAE output tensor ps to fill the buffer slot, not inflate the encoder input when sizes already align on one axis.

3. What Was Wrong at the v2.3.1 Milestone (Before 1e2f758a)

Stage Commit State
v2.3 shipped 24aefab9 Forge tiled VAE + end-align in tiled_scale_multidim. Multi-tile edge gaps fixed; single-tile full-axis pass[1] still NaN.
v2.3.1 — ControlNet cache (1cf51f90) 1cf51f90 ControlNet tile cache rebuild after canvas realign. No change to VAE NaN logic. NaN still reproduced on 1086×954 img2img.
v2.3.1 — NaN attempt (019e419d) 019e419d Replicate-pad encoder input in encode_fn. Fixed some H (966) but ** broke H=954** with narrow/size errors. End-align still present.
v2.3.1 — NaN shipped (1e2f758a) 1e2f758a Remove encoder-input pad; remove end-align; replicate-pad trailing edge of ps.

At v2.3.1 (after 1cf51f90, before 1e2f758a), the code still had end-align + encoder-input padding — the worst combination for mixed image heights.

4. Countermeasure (Final Fix in 1e2f758a)

  1. Delete end-align loop in tiled_scale_multidim (shifting upscaled[d]).
  2. Delete replicate padding in encode_fn (encoder input).
  3. Add after ps = function(s_in): compute per-axis gap between buffer slot end and (upscaled[d] + ps.shape[d+2]) for last_axis[d] tiles; replicate-pad ps on the trailing edge only so it exactly fills the slot before mask blending.

Properties:

  • Leading latent rows stay covered when one tile spans full height (pass[1] / H=966).
  • No encoder input inflation → H=954 stays 119×120 with no overshoot.
  • Replicate mode repeats the last latent row/col — neutral for edge pixels, avoids NaN.

Decode path is unchanged (decode uses downscale=True and different pass sizes; this bug was encode-specific).

5. Added / Changed Code (Full Text from 1e2f758a)

5-1. tiled_scale_multidim — removed end-align, added trailing replicate pad

Removed (was between ps = function(...) and mask = torch.ones_like(ps)):

            # End-align the last tile when VAE floor() leaves the output edge
            # uncovered (e.g. pixel width 2325 -> latent 291: last tile fills
            # 286-289 only, index 290 stays out_div==0 -> 0/0 = NaN).
            for d in range(dims):
                if last_axis[d] and upscaled[d] + mask.shape[d + 2] < out.shape[d + 2]:
                    upscaled[d] = max(0, out.shape[d + 2] - mask.shape[d + 2])

Added (current production code):

            ps = function(s_in).to(output_device)

            # Replicate-pad the trailing edge when the last tile's output is
            # smaller than the buffer slot. VAE returns floor(N/8) but the
            # buffer is sized round(N/8); for tiles whose last_axis fills to
            # the image edge, this leaves a 1-row/col gap (out_div==0 -> NaN).
            # Padding (not shifting via end-align) keeps the leading row
            # covered, which matters when a single tile spans the full axis
            # (e.g. pass[1] tile_y=1024 with image H=966): shifting would
            # leave row 0 as 0/0=NaN. F.pad expects (left_lastdim, right_lastdim,
            # ..., left_firstdim, right_firstdim), so build from the last spatial
            # dim backward.
            pad_amounts = []
            for d in range(dims - 1, -1, -1):
                if last_axis[d]:
                    gap = out.shape[d + 2] - (upscaled[d] + ps.shape[d + 2])
                    pad_amounts.extend([0, max(0, gap)])
                else:
                    pad_amounts.extend([0, 0])
            if any(p > 0 for p in pad_amounts):
                ps = torch.nn.functional.pad(ps, pad_amounts, mode="replicate")

            mask = torch.ones_like(ps)

5-2. _encode_tiled → encode_fn — removed encoder-input padding

Removed from inside encode_fn (after a = a.to(dtype=..., device=...)):

        # SD1.5/SDXL VAE encoder uses 3x asymmetric-padded stride-2 conv (pad=(0,1,0,1)).
        # Inputs with H or W not divisible by 8 produce floor(N/8)-sized latent, but
        # tiled_scale's output buffer is sized round(N/8). For a single tile spanning
        # the full input axis (e.g. pass[1] tile_y=1024 with image H=966), the
        # end-align branch shifts the (120-row) latent down by 1 row, leaving row 0
        # as out_div==0 -> NaN. Pad to next multiple of 8 with edge replication so the
        # VAE output matches the round-based buffer size.
        _, _, ah, aw = a.shape
        pad_h = (-ah) % 8
        pad_w = (-aw) % 8
        if pad_h or pad_w:
            a = torch.nn.functional.pad(a, (0, pad_w, 0, pad_h), mode="replicate")

**Current `encode_f...

Read more
Loading

v2.3.1 -

v2.3.1 -

Choose a tag to compare

View all tags
@ussoewwin ussoewwin released this 26 Jun 19:49
v2.3.1
37570af

Table of contents

  1. Error content (symptoms, log, traceback)
  2. Essential root cause
  3. Countermeasure (fix design)
  4. Full added/changed code (no omissions)
  5. Meaning of each change
  6. Related code you must understand
  7. Timeline: why 240×252 becomes 241×252 and batches 5→7
  8. Why set_controlnet_tensors_for_size alone was not enough
  9. Verification checklist
  10. Approaches that do not fix this bug

1. Error content (symptoms, log, traceback)

1-1. When it happens

Item Typical value
Pipeline img2img upscale with MultiDiffusion (Tiled Diffusion)
ControlNet Enabled (e.g. tile_resample / tile preprocessor)
Sampler extras Noise Inversion (Euler or other k-diff path)
VAE Forge Tiled VAE enabled (tiled encode at job start)
VRAM target ~16 GB — other fixes (bookend VAE, per-tile UNet) already applied

The job often survives tiled VAE encode and prints MultiDiffusion progress. The crash appears during Noise Inversion or the first tiled UNet step after the latent canvas is realigned to match init_latent.

1-2. Console log sequence (representative)

Before the exception, logs typically show:

[Tiled VAE]: encode ... (Forge 3-pass tiled encode completes)
[Tiled Diffusion]: Realign latent canvas 240x252 -> 241x252

Interpretation:

  • 240×252 — latent canvas size derived from UI / AbstractDiffusion init (pixel_to_latent_*, grid bbox split).
  • 241×252 — actual init_latent spatial size after Forge tiled encode (ceil rounding on width or height).
  • After realign, batched_bboxes is recomputed → more tiles and more batch groups (e.g. 5 → 7).

Then Python raises:

IndexError: list index out of range

1-3. Stack trace (essential frames)

File ".../tile_methods/multidiffusion.py", line 197, in sample_one_step
    self.switch_controlnet_tensors(batch_id, N, tb, tile_offset=0)
File ".../tile_methods/abstractdiffusion.py", line 699, in switch_controlnet_tensors
    batch_tiles = self.control_tensor_batch[param_id][batch_id]
IndexError: list index out of range

Failing expression: self.control_tensor_batch[param_id][batch_id]

  • param_id — index over ControlNet units (0 … number of enabled ControlNets − 1). Usually valid.
  • batch_id — index into the per-batch tile cache, one entry per element of self.batched_bboxes.
  • Failure mode: loop runs for batch_id, bboxes in enumerate(self.batched_bboxes) with batch_id ∈ {0,…,6} but control_tensor_batch[param_id] still has length 5 (built at init time against the old grid).

So this is not a ControlNet model bug or CUDA OOM. It is a stale Python list whose length no longer matches the rebuilt tile grid.

2. Essential root cause

2-1. Three parallel data structures

MultiDiffusion + ControlNet maintains three related structures:

Structure Role When built / updated (before fix)
org_control_tensor_batch Full-resolution ControlNet hint tensor(s) copied from control_params[].hint_cond at init prepare_controlnet_tensors() once at init_controlnet
batched_bboxes Latent-space tile grid: list of batches, each batch a list of BBox init_grid_bbox() at setup; rebuilt in _rebuild_latent_canvas() when (H,W) changes
control_tensor_batch Pre-cropped hint per tile batch, indexed by [param_id][batch_id] prepare_controlnet_tensors() only — uses batched_bboxes at init time

Invariant required for correctness:

len(control_tensor_batch[param_id]) == len(batched_bboxes)   for every param_id

Before commit 1cf51f90: _rebuild_latent_canvas() updated self.h, self.w, weights, and batched_bboxes, but did not rebuild control_tensor_batch or control_tensor_custom.

org_control_tensor_batch remained valid (full hint unchanged). Only the derived per-tile cache went stale.

2-2. Why canvas size changes after init

Forge tiled VAE encode returns init_latent whose H/W can differ by ±1 from the UI-derived canvas:

  • Integration uses pixel_to_latent_w = ceil(width/8) for width and floor for height in places; Forge encode may ceil latent dimensions.
  • Example: pixel width maps to latent W 291 on encode, while init canvas used 290 → after realign, 240 → 241 in latent rows/cols depending on image geometry and tile grid.

Any +1 change on an axis can add extra tiles along that axis → num_batches increases.

2-3. Call chain at crash

multidiffusion.sample_one_step(x_in, ...)
  N, C, H, W = x_in.shape
  if (H, W) != (self.h, self.w):
      self._rebuild_latent_canvas(H, W)          # batched_bboxes grows (e.g. 5 → 7 batches)
      if self.enable_controlnet:
          self.set_controlnet_tensors_for_size(H, W)   # updates global hint_cond only
  ...
  for batch_id, bboxes in enumerate(self.batched_bboxes):   # batch_id = 5, 6, ...
      self.switch_controlnet_tensors(batch_id, ...)
          self.control_tensor_batch[param_id][batch_id]   # IndexError: only 5 entries

Essence in one sentence: the tile iteration space was refreshed, but the ControlNet tile cache was still sized for the old iteration space.

3. Countermeasure (fix design)

3-1. Goal

After every _rebuild_latent_canvas(), recompute control_tensor_batch and control_tensor_custom from the unchanged source of truth org_control_tensor_batch, using the current batched_bboxes / custom_bboxes, with the same cropping logic as prepare_controlnet_tensors().

3-2. Why not call prepare_controlnet_tensors(refresh=True)?

prepare_controlnet_tensors() early-returns if caches already exist unless refresh=True, and it re-reads hints from latest_network.control_params. Rebuilding from org_control_tensor_batch is safer because:

  • Canvas rebuild can happen mid-job; network hooks may have mutated hint_cond via set_controlnet_tensors_for_size.
  • org_control_tensor_batch is explicitly kept as the full-resolution snapshot at init.
  • Logic is duplicated intentionally to mirror init cropping (_crop_controlnet_tile) without side effects on ControlNet global state.

3-3. Implementation summary

  1. Add _rebuild_controlnet_tile_cache() — same nested loops as prepare_controlnet_tensors body, source = org_control_tensor_batch.
  2. Call it at the end of _rebuild_latent_canvas(), after batched_bboxes / custom bbox scaling is final.
  3. No-op if ControlNet disabled or org_control_tensor_batch empty.

This restores the invariant len(control_tensor_batch[i]) == len(batched_bboxes).

4. Full added/changed code (no omissions)

4-1. Hook inside _rebuild_latent_canvas (lines 230–235)

These lines are inserted after the realign print and before return True:

        print(f'[Tiled Diffusion] Realign latent canvas {old_h}x{old_w} -> {self.h}x{self.w}')
        # batched_bboxes / custom_bboxes have changed; rebuild the per-tile ControlNet
        # cache so switch_controlnet_tensors does not IndexError when the new grid has
        # more batches than the cache built at init_controlnet time.
        self._rebuild_controlnet_tile_cache()
        return True

4-2. New method _rebuild_controlnet_tile_cache (lines 237–275)

Full method as committed:

    def _rebuild_controlnet_tile_cache(self) -> None:
        """Rebuild control_tensor_batch / control_tensor_custom for the current
        batched_bboxes / custom_bboxes, using org_control_tensor_batch (preserved
        full-resolution hint from init_controlnet) as the source of truth.

        Required when _rebuild_latent_canvas changes the tile grid (e.g. Forge
        tiled encode returns a ceil-rounded latent that is one row/col larger
        than the UI canvas). Without this, switch_controlnet_tensors indexes
        past the original control_tensor_batch length and raises IndexError."""
        if not getattr(self, 'enable_controlnet', False):
            return
        org_tensors = getattr(self, 'org_control_tensor_batch', None)
        if not org_tensors:
            return
        self.control_tensor_batch = []
        # control_tensor_custom is a per-param list of per-bbox tile lists; reset
        # only when we actually rebuild, so init_controlnet's empty list survives
        # the no-ControlNet case.
        self.control_tensor_custom = []
        for i in range(len(org_tensors)):
            control_tile_list = []
            control_tensor = org_tensors[i]
            for bboxes in self.batched_bboxes:
                single_batch_tensors = []
                for bbox in bboxes:
                    single_batch_tensors.append(self._crop_controlnet_tile(control_tensor, bbox))
                control_tile = torch.cat(single_batch_tensors, dim=0)
                if self.control_tensor_cpu:
                    control_tile = control_tile.cpu()
                control_tile_list.append(control_tile)
            self.control_tensor_batch.append(control_tile_list)
            if len(self.custom_bboxes) > 0:
                custom_control_tile_list = []
       ...
Read more
Loading