Skip to content

v3.0.4 - Early NumPy Validation & Self-Healing against False Torch GPU Startup Crash

Choose a tag to compare

@ussoewwin ussoewwin released this 13 Oct 16:07
· 21 commits to main since this release

① Overview of the NumPy Early Validation & Startup Deadlock Fix

1. Background (State before the change)

In Python 3.14 environments, launching A1111 via webui-user.bat could abruptly abort during startup with the following error:

RuntimeError: Torch is not able to use GPU; add --skip-torch-cuda-test to COMMANDLINE_ARGS variable to disable this check

Despite the error message pointing to GPU or CUDA issues, the underlying GPU hardware, NVIDIA drivers, and PyTorch (torch==2.14.0+cu132) installation were completely operational.

2. Root Cause Analysis (Technical Deep Dive)

  1. Destructive Unconditional Reinstallation on Every Launch:
    Previously in modules/launch_utils.py, prepare_environment() unconditionally ran:

    • pip uninstall numpy -y followed by pip install --no-cache-dir numpy==2.5.3
    • pip uninstall scipy -y followed by pip install --no-cache-dir scipy==1.16.1
    • A secondary --force-reinstall --no-deps --no-cache-dir numpy==2.5.3 after extension installers.
      If any startup process was closed, killed, or interrupted while running these uninstalls, the venv/Lib/site-packages/numpy directory was left in a half-uninstalled state (e.g. version.py or _distributor_init.py remained while __init__.py and _core/ were removed).
  2. PEP 420 Namespace Package Trapping PyTorch Imports:
    Under Python 3.3+ (PEP 420), any directory without an __init__.py is resolved by CPython as an implicit Namespace Package (<module 'numpy' (namespace)>).
    When launch.py executed its startup check import torch; assert torch.cuda.is_available():

    • torch/__init__.py imports torch.testing (line 2720), which imports torch.testing._comparison.
    • torch.testing._comparison contains:
      try:
          import numpy as np
          HAS_NUMPY = True
      except ModuleNotFoundError:
          HAS_NUMPY = False
      if HAS_NUMPY:
          _IS_DATACLASS_SKIP_TYPES = (*_IS_DATACLASS_SKIP_TYPES, np.ndarray, np.generic)
    • Because site-packages/numpy existed without __init__.py, import numpy as np did not raise ModuleNotFoundError and succeeded as a namespace package.
    • Accessing np.ndarray immediately threw AttributeError: module 'numpy' has no attribute 'ndarray'.
    • Furthermore, torchvision.utils unconditionally executes import numpy as np, which also crashed with ModuleNotFoundError / AttributeError.
  3. Startup Ordering Deadlock:
    In modules/launch_utils.py, the check_run_python("import torch; assert torch.cuda.is_available()") check was positioned before line 735 (where numpy was reinstalled).
    Because check_run_python captures standard error and only checks the process exit code, the AttributeError: module 'numpy' has no attribute 'ndarray' from import torch caused the GPU check to return non-zero, immediately raising RuntimeError: Torch is not able to use GPU. As a result, execution aborted before it could ever reach the NumPy reinstallation block, creating a permanent startup deadlock.


3. What Was Changed

  1. Pre-Check Self-Healing & Early Validation:

    • Added an early validation step in modules/launch_utils.py before _apply_torch_stack_pip_constraint() and the torch.cuda.is_available() test.
    • It verifies whether NumPy is cleanly importable: check_run_python("import numpy; assert hasattr(numpy, 'ndarray') and numpy.__version__ == '2.5.3'").
    • If missing or corrupted, any broken site-packages/numpy residue directory is safely purged via shutil.rmtree() to prevent PEP 420 namespace package poisoning, followed by a clean reinstall of numpy==2.5.3.
  2. Elimination of Redundant Startup Reinstall Cycles:

    • Guarded the startup numpy==2.5.3 and scipy==1.16.1 installation blocks with check_run_python assertions.
    • If numpy==2.5.3 (with valid ndarray) and scipy==1.16.1 are already present in the environment, the destructive pip uninstall and --no-cache-dir network downloads are skipped entirely.
    • The post-extension re-pin is similarly guarded to run only if an extension installer drifted NumPy away from 2.5.3.

4. Verification Results

  • Verified that clean/repaired numpy==2.5.3 enables import torch, torchvision and torch.cuda.is_available() to succeed without errors.
  • Verified that consecutive startups skip redundant package uninstalls, cutting startup latency and eliminating the risk of partial-uninstall filesystem corruption.
  • Verified repository documentation and changelog alignment across md/CHANGELOG.md and README.md.

② Modified Files

File Change
modules/launch_utils.py Added pre-check NumPy validation/self-healing before PyTorch GPU test; guarded redundant package reinstalls
md/CHANGELOG.md Added v3.0.4 release notes and updated header
README.md Updated NumPy pin documentation reference to 2.5.3