Torch nightly from 20250630 and onwards enables (forced) logging/debugging to INFO on any application.

[Panchovix]

Hi there, thanks for all the great work.

I have been always installing pytorch nightly.

After updating this week I've noticed that logging is enabled on my app (reForge) forcefully. This also happens, for example, when using ComfyUI.

I'm installing with

pip install torch torchvision --index-url https://download.pytorch.org/whl/nightly/cu129

And I get this log

Loading model /naiXLVpred102d_colorized.safetensors [2a7fc06348] (1 of 1)
Loading weights [2a7fc06348] from /naiXLVpred102d_colorized.safetensors
CHv1.8.13: Set Proxy: 
INFO:root:model weight dtype torch.float16, manual cast: None
INFO:root:model_type V_PREDICTION
INFO:root:Using pytorch attention in VAE
INFO:root:Using pytorch attention in VAE
INFO:root:VAE load device: cuda:0, offload device: cuda:0, dtype: torch.bfloat16
INFO:root:Conditional stage model device compatibility check is disabled.
INFO:root:Requested to load SDXLClipModel
INFO:root:loaded completely 9.5367431640625e+25 1749.853515625 True
INFO:root:CLIP/text encoder model load device: cuda:0, offload device: cuda:0, current: cuda:0, dtype: torch.float16
WARNING:root:clip missing: ['clip_l.text_projection', 'clip_l.logit_scale']
loaded diffusion model directly to GPU
INFO:root:Requested to load SDXL
INFO:root:loaded completely 9.5367431640625e+25 4897.0483474731445 True
Loading VAE weights specified in settings: /models/VAE/sdxlVaeAnimeTest_alpha67500.safetensors
Reloading VAE
Loading VAE weights specified in settings: /models/VAE/sdxlVaeAnimeTest_alpha67500.safetensors
INFO:root:Requested to load SDXLClipModel
INFO:root:loaded completely 9.5367431640625e+25 2521.853515625 True
Model /naiXLVpred102d_colorized.safetensors [2a7fc06348] loaded in 2.3s (memory cleanup: 0.5s, forge load real models: 1.3s, load VAE: 0.2s, calculate empty prompt: 0.1s).
Memory change: 0.00 MB (10.62 GB total)
2025-07-14 20:41:01,152 - ControlNet - INFO - ControlNet UI callback registered.
Running on local URL:  https://0.0.0.0:8280

And then for every subsequent generation or anything that touches the application/API, more logging/debug is outputted. i.e.:

INFO:httpx:HTTP Request: POST https://localhost:8280/api/predict "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://localhost:8280/api/predict "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://localhost:8280/api/predict "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://localhost:8280/api/predict "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://localhost:8280/api/predict "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://localhost:8280/reset "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://localhost:8280/reset "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://localhost:8280/reset "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://localhost:8280/reset "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://localhost:8280/reset "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://localhost:8280/api/predict "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://localhost:8280/api/predict "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://localhost:8280/reset "HTTP/1.1 200 OK"
INFO:httpx:HTTP Request: POST https://localhost:8280/reset "HTTP/1.1 200 OK"

But when I install with

pip install torch==2.9.0.dev20250629 torchvision --index-url https://download.pytorch.org/whl/nightly/cu129

Logging/debugging is not forced enabled, and it looks like this:

Loading model /naiXLVpred102d_colorized.safetensors [2a7fc06348] (1 of 1)
Loading weights [2a7fc06348] from /naiXLVpred102d_colorized.safetensors
CHv1.8.13: Set Proxy: 
WARNING:root:clip missing: ['clip_l.text_projection', 'clip_l.logit_scale']
loaded diffusion model directly to GPU
Loading VAE weights specified in settings: /models/VAE/sdxlVaeAnimeTest_alpha67500.safetensors
Reloading VAE
Loading VAE weights specified in settings: /models/VAE/sdxlVaeAnimeTest_alpha67500.safetensors
Model /naiXLVpred102d_colorized.safetensors [2a7fc06348] loaded in 2.3s (memory cleanup: 0.5s, forge load real models: 1.4s, load VAE: 0.2s, calculate empty prompt: 0.1s).
Memory change: 0.00 MB (10.76 GB total)
2025-07-14 20:38:49,048 - ControlNet - INFO - ControlNet UI callback registered.
Running on local URL:  https://0.0.0.0:8280

This happens with either torch+cu128 or torch+cu129.

Is there maybe a setting or env variable to disable this? As normal env variables or commands to set debugging no "None" doesn't work.

Thanks.

cc @ezyang @gchanan @zou3519 @kadeng @msaroufim

[ezyang]

I'm asking Codex if it can find the culprit diff

[malfet]

@atalman please make sure that 2.8 is not affected...

[albanD]

From triage meeting: adding 2.8 milestone to double check it is not affected.

[ezyang]

From Codex: (full transcript https://chatgpt.com/s/cd_687ef3ca248c8191b07b3739f235b411 )

---

The excessive output is triggered when importing the experimental distributed checkpointing module.
In commit dd3e717 (“Add async checkpointing impl to experimental checkpointer and add a builder API”), the new file torch/distributed/checkpoint/_experimental/staging.py initializes the root logger and sets its level to INFO:

logger = getLogger()
logger.setLevel(logging.INFO)

Since getLogger() with no name returns the root logger, this raises the root logging level to INFO whenever the module is imported, causing other libraries (e.g. httpx) to emit INFO-level logs even if the application didn’t configure logging. This behavior was introduced by the above commit (dated July 3 2025).

Thus, importing torch.distributed.checkpoint._experimental.staging after this commit leads to global logging being enabled unintentionally. The prior nightly (before this commit) lacked this root-level modification, so no logs appeared.

---

@teja-rao @pradeepfn please take a look. #156927 The timeline seems to line up and I confirm the problematic lines Codex pointed out:

diff --git a/torch/distributed/checkpoint/_experimental/staging.py b/torch/distributed/checkpoint/_experimental/staging.py
new file mode 100644
index 00000000000..55e4c15921a
--- /dev/null
+++ b/torch/distributed/checkpoint/_experimental/staging.py
@@ -0,0 +1,216 @@
+"""
+Experimental staging module for PyTorch Distributed Checkpointing.
+
+This module provides advanced staging capabilities for checkpoints including:
+- Asynchronous staging using ThreadPoolExecutor
+- Pinned memory allocation for faster CPU-GPU transfers
+- Shared memory support for multi-process scenarios
+- Non-blocking CUDA operations with stream synchronization
+- Caching of frequently used storages for efficient memory management
+- Automatic resource cleanup and memory management
+
+Classes:
+    CheckpointStager: Abstract base class defining the staging interface
+    StagingOptions: Configuration dataclass for staging behavior
+    DefaultStager: Default implementation with comprehensive staging features
+"""
+
+import abc
+import logging
+from concurrent.futures import Future, ThreadPoolExecutor
+from dataclasses import dataclass
+from logging import getLogger
+from typing import Any, TypeVar, Union
+
+import torch
+from torch.distributed.checkpoint._state_dict_stager import StateDictStager
+
+from .types import STATE_DICT
+
+
+T = TypeVar("T")
+
+logger = getLogger()
+logger.setLevel(logging.INFO)

[jilimeta]

Claude answer (no coding) - not sure if it's useful https://claude.ai/share/9790310b-8406-41b2-9eb6-f445c18f257f