[Bug]: Windows GPU OOM error

[RWL-Dittrich]

Steps to Reproduce

I'm trying to cluster two mac mini's and an RTX 4090 24GB together to run GPT-OSS 120b.
Somehow the RTX 4090 gets assigned more memory than it can handle.

I ran parallax install and parallax check to make sure my environment is up-to-date and correctly setup.

Expected Behavior

I expect the model to be scaled correctly according to the amount of memory a node has

Actual Behavior

Parallax crashes with the following stacktrace

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1014.00 MiB. GPU 0 has a total capacity of 23.99 GiB of which 0 bytes is free. Including non-PyTorch memory, this process has 17179869184.00 GiB memory in use. Of the allocated memory 31.04 GiB is allocated by PyTorch, and 126.87 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Nov 10 09:03:46.891 [�[1m�[31mERROR   �[0m] launch.py:175             CUDA out of memory. Tried to allocate 1014.00 MiB. GPU 0 has a total capacity of 23.99 GiB of which 0 bytes is free. Including non-PyTorch memory, this process has 17179869184.00 GiB memory in use. Of the allocated memory 31.04 GiB is allocated by PyTorch, and 126.87 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Traceback (most recent call last):
  File "/root/parallax/src/parallax/launch.py", line 127, in <module>
    executor = Executor.create_from_args(args, gradient_server=gradient_server)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/src/parallax/server/executor.py", line 289, in create_from_args
    return cls(**create_executor_config(args, gradient_server))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/src/parallax/server/executor.py", line 116, in __init__
    self.model_runner, self.config, self.tokenizer = initialize_sgl_model_runner(
                                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/src/parallax/sglang/model_runner.py", line 296, in initialize_sgl_model_runner
    model_runner = ParallaxModelRunner(
                   ^^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/src/parallax/sglang/model_runner.py", line 76, in __init__
    super().__init__(
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 312, in __init__
    self.initialize(min_per_gpu_memory)
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 384, in initialize
    self.load_model()
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 739, in load_model
    self.model = get_model(
                 ^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/model_loader/__init__.py", line 28, in get_model
    return loader.load_model(
           ^^^^^^^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/model_loader/loader.py", line 590, in load_model
    model = _initialize_model(
            ^^^^^^^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/model_loader/loader.py", line 262, in _initialize_model
    return model_class(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/models/gpt_oss.py", line 586, in __init__
    self.model = GptOssModel(
                 ^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/models/gpt_oss.py", line 507, in __init__
    self.layers, self.start_layer, self.end_layer = make_layers(
                                                    ^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/utils/common.py", line 560, in make_layers
    + get_offloader().wrap_modules(
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/utils/offloader.py", line 36, in wrap_modules
    return list(all_modules_generator)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/utils/common.py", line 562, in <genexpr>
    layer_fn(idx=idx, prefix=add_prefix(idx, prefix))
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/models/gpt_oss.py", line 509, in <lambda>
    lambda idx, prefix: decoder_layer_type(
                        ^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/models/gpt_oss.py", line 416, in __init__
    self.mlp = GptOssSparseMoeBlock(
               ^^^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/models/gpt_oss.py", line 139, in __init__
    self.experts = experts_type(
                   ^^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 212, in __init__
    self.quant_method.create_weights(
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/layers/quantization/mxfp4.py", line 313, in create_weights
    torch.zeros(
  File "/root/parallax/venv/lib/python3.12/site-packages/torch/utils/_device.py", line 103, in __torch_function__
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1014.00 MiB. GPU 0 has a total capacity of 23.99 GiB of which 0 bytes is free. Including non-PyTorch memory, this process has 17179869184.00 GiB memory in use. Of the allocated memory 31.04 GiB is allocated by PyTorch, and 126.87 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Nov 10 09:03:46.892 [�[1m�[32mINFO    �[0m] server.py:723             Leave scheduler: 12D3KooWGG6rkFt9c33wriGrnDquef4bkhWSwZTzRfmBue2jCpMy
[rank0]:[W1110 09:03:53.902193591 ProcessGroupNCCL.cpp:1538] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
[INFO] Successfully joined the distributed inference cluster.

Version

7ff90da

Environment & Context

• I'm using the latest version.
• I have searched existing issues.

[TianyiZhao1437]

Hi, I have not been able to reproduce this problem on 4090. Could you help add --log-level DEBUG in the parallax join command and paste the log here?
parallax join --log-level DEBUG

[RWL-Dittrich]

I just tried fully reinstalling the latest version of Parallax, running parallax install and parallax check. Those commands all give me correct results so my environment seems to be set up correctly.

For reference, I am running a cluster of two Apple Mac Mini's (M4 Pro) with 64 gigs of Unified memory each and I am trying to add the 4090 to the cluster too.

I re-ran the command with DEBUG logging enabled and got this full output:

PS C:\Users\installer> parallax join --log-level DEBUG
Nov 12 11:05:23.288 [INFO    ] cli.py:260                Scheduler address: auto
Nov 12 11:05:23.288 [INFO    ] cli.py:99                 Running command: /root/parallax/venv/bin/python3 /root/parallax/src/parallax/launch.py --max-num-tokens-per-batch 4096 --max-sequence-length 2048 --max-batch-size 8 --kv-block-size 1024 --scheduler-addr auto --log-level DEBUG
Nov 12 11:05:27.264 [DEBUG   ] launch.py:48              args: Namespace(host='localhost', port=3000, node_chat_port=3002, initial_peers=[], scheduler_addr='auto', relay_servers=[], tcp_port=0, udp_port=0, announce_maddrs=[], dht_prefix='gradient', notify_url=None, model_path=None, max_sequence_length=2048, param_hosting_ratio=0.65, kv_cache_ratio=0.25, start_layer=None, end_layer=None, dtype='bfloat16', kv_cache_memory_fraction=0.8, kv_block_size=1024, enable_prefix_cache=False, max_batch_size=8, max_num_tokens_per_batch=4096, prefill_priority=0, micro_batch_ratio=2, scheduler_wait_ms=500, request_timeout_s=600, attention_backend='flashinfer', moe_runner_backend='auto', tp_size=1, nccl_port=None, log_level='DEBUG', verbose=False, use_hfcache=False)
Nov 12 11:05:27.266 [DEBUG   ] launch.py:59              executor_input_addr: ipc:///tmp/tmpbafrdmm6
Nov 12 11:05:27.266 [DEBUG   ] launch.py:60              executor_output_addr: ipc:///tmp/tmp47ofc8ca
Nov 12 11:05:27.266 [DEBUG   ] launch.py:61              nccl_port: 4792
Nov 12 11:05:27.266 [DEBUG   ] server.py:251             manual_layer_assignment: False
Nov 12 11:05:30.481 [INFO    ] server.py:301             Found scheduler peer id: 12D3KooWBwM4Rh3Fi5MwEVAn32gmANfepqtmvsJHg9gXFMpgC5yH
Nov 12 11:05:30.481 [INFO    ] server.py:316             Lattica built successfully
Nov 12 11:05:31.055 [INFO    ] server.py:343             Join scheduler response: {'node_id': '12D3KooWNzpMeg1faNAW1wKdgApz5fp9qWf5QDLE11tBwgZn6JUt', 'model_name': 'openai/gpt-oss-120b', 'start_layer': 0, 'end_layer': 9}
Nov 12 11:05:31.060 [DEBUG   ] server.py:591             Heartbeat: Node 12D3KooWNzpMeg1faNAW1wKdgApz5fp9qWf5QDLE11tBwgZn6JUt... Model: openai/gpt-oss-120b, Layers: [0, 9)
Nov 12 11:05:31.371 [DEBUG   ] launch.py:135             Start Executor with start_layer: 0, end_layer: 9
Nov 12 11:05:31.609 [DEBUG   ] executor.py:111           Executor initializing on device: cuda
/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/environ.py:272: UserWarning: Environment variable SGL_ENABLE_JIT_DEEPGEMM is deprecated, please use SGLANG_ENABLE_JIT_DEEPGEMM
  warnings.warn(
Fetching 14 files: 100%|██████████| 14/14 [00:00<00:00, 283672.73it/s]
Nov 12 11:05:34.936 [DEBUG   ] executor.py:119           Initializing CUDA model runner for repo=openai/gpt-oss-120b, layers=[0, 9)
Nov 12 11:05:34.967 [INFO    ] model_runner.py:248       Downloading model with selective weight files for layers [0, 9)
Nov 12 11:05:34.967 [DEBUG   ] selective_download.py:59  Downloading model metadata for openai/gpt-oss-120b
INFO:     Started server process [476]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://localhost:3000 (Press CTRL+C to quit)
Nov 12 11:05:35.105 [DEBUG   ] selective_download.py:66  Downloaded model metadata to /root/.cache/huggingface/hub/models--openai--gpt-oss-120b/snapshots/b5c939de8f754692c1647ca79fbf85e8c1e70f8a
Nov 12 11:05:35.105 [DEBUG   ] selective_download.py:70  Determining required weight files for layers [0, 9)
Nov 12 11:05:35.106 [DEBUG   ] weight_filter_utils.py:158 Determined 9 weight files needed for layers [0, 9)
Nov 12 11:05:35.106 [INFO    ] selective_download.py:89  Downloading 9 weight files
Nov 12 11:05:35.106 [DEBUG   ] selective_download.py:92  Downloading model-00001-of-00014.safetensors
Nov 12 11:05:35.210 [DEBUG   ] selective_download.py:92  Downloading model-00002-of-00014.safetensors
Nov 12 11:05:35.315 [DEBUG   ] selective_download.py:92  Downloading model-00004-of-00014.safetensors
Nov 12 11:05:35.420 [DEBUG   ] selective_download.py:92  Downloading model-00005-of-00014.safetensors
Nov 12 11:05:35.595 [DEBUG   ] selective_download.py:92  Downloading model-00006-of-00014.safetensors
Nov 12 11:05:35.708 [DEBUG   ] selective_download.py:92  Downloading model-00009-of-00014.safetensors
Nov 12 11:05:35.812 [DEBUG   ] selective_download.py:92  Downloading model-00010-of-00014.safetensors
Nov 12 11:05:35.913 [DEBUG   ] selective_download.py:92  Downloading model-00012-of-00014.safetensors
Nov 12 11:05:36.024 [DEBUG   ] selective_download.py:92  Downloading model-00014-of-00014.safetensors
Nov 12 11:05:36.130 [DEBUG   ] selective_download.py:101 Downloaded weight files for layers [0, 9)
Nov 12 11:05:36.942 [INFO    ] model_config.py:852       Downcasting torch.float32 to torch.bfloat16.
Nov 12 11:05:36.942 [WARNING ] model_config.py:691       mxfp4 quantization is not fully optimized yet. The speed can be slower than non-quantized models.
Nov 12 11:05:36.943 [INFO    ] model_config.py:852       Downcasting torch.float32 to torch.bfloat16.
Nov 12 11:05:36.943 [WARNING ] model_config.py:691       mxfp4 quantization is not fully optimized yet. The speed can be slower than non-quantized models.
Nov 12 11:05:36.943 [DEBUG   ] model_runner.py:299       model_start_layer: 0
Nov 12 11:05:36.943 [DEBUG   ] model_runner.py:300       model_end_layer: 9
Nov 12 11:05:36.943 [INFO    ] model_runner.py:94        Init torch distributed begin.
Nov 12 11:05:37.080 [DEBUG   ] parallel_state.py:1413    world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:4792 backend=nccl
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
Nov 12 11:05:37.351 [INFO    ] model_runner.py:192       Init torch distributed ends. mem usage=0.00 GB
Nov 12 11:05:38.519 [INFO    ] model_runner.py:671       Load weight begin. avail mem=22.37 GB
`torch_dtype` is deprecated! Use `dtype` instead!
Nov 12 11:05:41.036 [ERROR   ] launch.py:187             Executor error: CUDA out of memory. Tried to allocate 1014.00 MiB. GPU 0 has a total capacity of 23.99 GiB of which 0 bytes is free. Including non-PyTorch memory, this process has 17179869184.00 GiB memory in use. Of the allocated memory 53.30 GiB is allocated by PyTorch, and 202.24 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Traceback (most recent call last):
  File "/root/parallax/src/parallax/launch.py", line 151, in <module>
    executor = Executor.create_from_args(args, gradient_server=gradient_server)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/src/parallax/server/executor.py", line 307, in create_from_args
    return cls(**create_executor_config(args, gradient_server))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/src/parallax/server/executor.py", line 122, in __init__
    self.model_runner, self.config, self.tokenizer = initialize_sgl_model_runner(
                                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/src/parallax/sglang/model_runner.py", line 302, in initialize_sgl_model_runner
    model_runner = ParallaxModelRunner(
                   ^^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/src/parallax/sglang/model_runner.py", line 75, in __init__
    super().__init__(
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 312, in __init__
    self.initialize(min_per_gpu_memory)
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 384, in initialize
    self.load_model()
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 739, in load_model
    self.model = get_model(
                 ^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/model_loader/__init__.py", line 28, in get_model
    return loader.load_model(
           ^^^^^^^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/model_loader/loader.py", line 590, in load_model
    model = _initialize_model(
            ^^^^^^^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/model_loader/loader.py", line 262, in _initialize_model
    return model_class(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/models/gpt_oss.py", line 586, in __init__
    self.model = GptOssModel(
                 ^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/models/gpt_oss.py", line 507, in __init__
    self.layers, self.start_layer, self.end_layer = make_layers(
                                                    ^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/utils/common.py", line 560, in make_layers
    + get_offloader().wrap_modules(
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/utils/offloader.py", line 36, in wrap_modules
    return list(all_modules_generator)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/utils/common.py", line 562, in <genexpr>
    layer_fn(idx=idx, prefix=add_prefix(idx, prefix))
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/models/gpt_oss.py", line 509, in <lambda>
    lambda idx, prefix: decoder_layer_type(
                        ^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/models/gpt_oss.py", line 416, in __init__
    self.mlp = GptOssSparseMoeBlock(
               ^^^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/models/gpt_oss.py", line 139, in __init__
    self.experts = experts_type(
                   ^^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 212, in __init__
    self.quant_method.create_weights(
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/layers/quantization/mxfp4.py", line 313, in create_weights
    torch.zeros(
  File "/root/parallax/venv/lib/python3.12/site-packages/torch/utils/_device.py", line 103, in __torch_function__
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1014.00 MiB. GPU 0 has a total capacity of 23.99 GiB of which 0 bytes is free. Including non-PyTorch memory, this process has 17179869184.00 GiB memory in use. Of the allocated memory 53.30 GiB is allocated by PyTorch, and 202.24 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Nov 12 11:05:41.041 [ERROR   ] launch.py:199             CUDA out of memory. Tried to allocate 1014.00 MiB. GPU 0 has a total capacity of 23.99 GiB of which 0 bytes is free. Including non-PyTorch memory, this process has 17179869184.00 GiB memory in use. Of the allocated memory 53.30 GiB is allocated by PyTorch, and 202.24 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Traceback (most recent call last):
  File "/root/parallax/src/parallax/launch.py", line 151, in <module>
    executor = Executor.create_from_args(args, gradient_server=gradient_server)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/src/parallax/server/executor.py", line 307, in create_from_args
    return cls(**create_executor_config(args, gradient_server))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/src/parallax/server/executor.py", line 122, in __init__
    self.model_runner, self.config, self.tokenizer = initialize_sgl_model_runner(
                                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/src/parallax/sglang/model_runner.py", line 302, in initialize_sgl_model_runner
    model_runner = ParallaxModelRunner(
                   ^^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/src/parallax/sglang/model_runner.py", line 75, in __init__
    super().__init__(
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 312, in __init__
    self.initialize(min_per_gpu_memory)
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 384, in initialize
    self.load_model()
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 739, in load_model
    self.model = get_model(
                 ^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/model_loader/__init__.py", line 28, in get_model
    return loader.load_model(
           ^^^^^^^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/model_loader/loader.py", line 590, in load_model
    model = _initialize_model(
            ^^^^^^^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/model_loader/loader.py", line 262, in _initialize_model
    return model_class(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/models/gpt_oss.py", line 586, in __init__
    self.model = GptOssModel(
                 ^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/models/gpt_oss.py", line 507, in __init__
    self.layers, self.start_layer, self.end_layer = make_layers(
                                                    ^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/utils/common.py", line 560, in make_layers
    + get_offloader().wrap_modules(
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/utils/offloader.py", line 36, in wrap_modules
    return list(all_modules_generator)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/utils/common.py", line 562, in <genexpr>
    layer_fn(idx=idx, prefix=add_prefix(idx, prefix))
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/models/gpt_oss.py", line 509, in <lambda>
    lambda idx, prefix: decoder_layer_type(
                        ^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/models/gpt_oss.py", line 416, in __init__
    self.mlp = GptOssSparseMoeBlock(
               ^^^^^^^^^^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/models/gpt_oss.py", line 139, in __init__
    self.experts = experts_type(
                   ^^^^^^^^^^^^^
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 212, in __init__
    self.quant_method.create_weights(
  File "/root/parallax/venv/lib/python3.12/site-packages/sglang/srt/layers/quantization/mxfp4.py", line 313, in create_weights
    torch.zeros(
  File "/root/parallax/venv/lib/python3.12/site-packages/torch/utils/_device.py", line 103, in __torch_function__
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1014.00 MiB. GPU 0 has a total capacity of 23.99 GiB of which 0 bytes is free. Including non-PyTorch memory, this process has 17179869184.00 GiB memory in use. Of the allocated memory 53.30 GiB is allocated by PyTorch, and 202.24 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Nov 12 11:05:41.042 [INFO    ] http_server.py:500        Stopping HTTP server process...
Nov 12 11:05:41.042 [INFO    ] server.py:723             Leave scheduler: 12D3KooWNzpMeg1faNAW1wKdgApz5fp9qWf5QDLE11tBwgZn6JUt
[rank0]:[W1112 11:05:42.717221070 ProcessGroupNCCL.cpp:1538] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
/usr/lib/python3.12/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
[INFO] Successfully joined the distributed inference cluster.
PS C:\Users\installer>

[RWL-Dittrich]

I think my issue might be related to pull request #225 and Issue #188. But I can't easily test this unfortunately.

[TianyiZhao1437]

PR #228 is expected to fix the problem.

[RWL-Dittrich]

I just tested it, It's fixed 🥳