BFloat16 Unsupported on MPS (macOS Sequoia 15.1.1, Intel, AMD)

[SkyForceCoder]

🐛 Describe the bug

I am encountering the error TypeError: BFloat16 is not supported on MPS while using PyTorch with Apple's Metal Performance Shaders (MPS) backend. This issue arises across multiple projects, specifically Stable Diffusion WebUI Forge and ComfyUI, both of which depend on PyTorch for GPU acceleration.

Processor:

Model: Intel(R) Xeon(R) W-2140B CPU
Speed: 3.20 GHz
Cores: 8 cores (with Hyper-Threading enabled)

Memory (RAM):

Total RAM: 32 GB (34359738368 bytes)

GPU:

Model: Radeon Pro Vega 56
VRAM: 8 GB
Bus: PCIe (x16)
Metal Support: Metal 3 (supports Apple's graphics framework)

Projects where this issue occurs:

1. [Stable Diffusion WebUI Forge Issue #2399](macOS Sequoia 15.1.1 (intel, AMD) - with MPS and PyTorch (BFloat16 Unsupported) lllyasviel/stable-diffusion-webui-forge#2399)
2. [ComfyUI Issue #5829](ComfyUI Startup Log on macOS Sequoia 15.1.1 with MPS and PyTorch (BFloat16 Unsupported) Comfy-Org/ComfyUI#5829)

Steps to Reproduce:

1. Use macOS with an MPS-supported GPU.
2. Run ComfyUi or Forge that utilize BFloat16 operations during processing.
3. The error consistently appears during model sampling or inference:
   TypeError: BFloat16 is not supported on MPS

Observations:

• The error suggests that the MPS backend in PyTorch doesn't support BFloat16 precision.
• This issue is critical because many ML pipelines and UIs default to BFloat16 or use it for performance reasons.

System:
macOS : Sequoia (15.1.1) Intel, AMD

Request:

Can BFloat16 support be added to MPS in PyTorch, CPU is very very slow for this type of operations.

Versions

2.1.2
ALL VERSIONS

cc @seemethere @malfet @osalpekar @atalman @albanD @frank-wei @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10 @kulinseth @DenisVieriu97 @jhavukainen

[hvaara]

Your PyTorch version is very old. Does it work if you update?

[SkyForceCoder]

@hvaara

I see latest version is 2.5.1 but it's not instaling

% pip install torch==2.3.1
ERROR: Could not find a version that satisfies the requirement torch==2.3.1 (from versions: 1.11.0, 1.12.0, 1.12.1, 1.13.0, 1.13.1, 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1, 2.2.2)
ERROR: No matching distribution found for torch==2.3.1

% pip3 install torch torchvision torchaudio

Successfully installed torch-2.2.2 torchaudio-2.2.2 torchvision-0.17.2

% python -c "import torch; print(f'PyTorch Version: {torch.version}'); print(f'MPS Available: {torch.backends.mps.is_available()}')"

PyTorch Version: 2.2.2
MPS Available: True

Error

Traceback (most recent call last):
File "/Volumes/Spidy/GitHub/AI/stable-diffusion-webui-forge/modules_forge/main_thread.py", line 30, in work
self.result = self.func(*self.args, **self.kwargs)
File "/Volumes/Spidy/GitHub/AI/stable-diffusion-webui-forge/modules/txt2img.py", line 131, in txt2img_function
processed = processing.process_images(p)
File "/Volumes/Spidy/GitHub/AI/stable-diffusion-webui-forge/modules/processing.py", line 842, in process_images
res = process_images_inner(p)
File "/Volumes/Spidy/GitHub/AI/stable-diffusion-webui-forge/modules/processing.py", line 990, in process_images_inner
samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
File "/Volumes/Spidy/GitHub/AI/stable-diffusion-webui-forge/modules/processing.py", line 1387, in sample
samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
File "/Volumes/Spidy/GitHub/AI/stable-diffusion-webui-forge/modules/sd_samplers_kdiffusion.py", line 198, in sample
sampling_prepare(self.model_wrap.inner_model.forge_objects.unet, x=x)
File "/Volumes/Spidy/GitHub/AI/stable-diffusion-webui-forge/backend/sampling/sampling_function.py", line 383, in sampling_prepare
memory_management.load_models_gpu(
File "/Volumes/Spidy/GitHub/AI/stable-diffusion-webui-forge/backend/memory_management.py", line 684, in load_models_gpu
loaded_model.model_load(model_gpu_memory_when_using_cpu_swap)
File "/Volumes/Spidy/GitHub/AI/stable-diffusion-webui-forge/backend/memory_management.py", line 516, in model_load
bake_gguf_model(self.real_model)
File "/Volumes/Spidy/GitHub/AI/stable-diffusion-webui-forge/backend/memory_management.py", line 347, in bake_gguf_model
gguf_cls.bake(p)
File "/Volumes/Spidy/GitHub/AI/stable-diffusion-webui-forge/packages_3rdparty/gguf/quants.py", line 152, in bake
cls.bake_inner(parameter)
File "/Volumes/Spidy/GitHub/AI/stable-diffusion-webui-forge/packages_3rdparty/gguf/quants.py", line 310, in bake_inner
d = d.view(torch.float16).to(parameter.computation_dtype).view(torch.uint8)
TypeError: BFloat16 is not supported on MPS
BFloat16 is not supported on MPS

[hvaara]

I think you tried to install 2.3.1, not 2.5.1. Alternatively run pip3 install -U torch.

[SkyForceCoder]

@hvaara but for me latest version is 2.2.2 is downloading

% python --version
Python 3.10.14

% pip3 install --no-cache-dir -U torch torchvision torchaudio
Collecting torch
Downloading torch-2.2.2-cp310-none-macosx_10_9_x86_64.whl.metadata (25 kB)
Collecting torchvision
Downloading torchvision-0.17.2-cp310-cp310-macosx_10_13_x86_64.whl.metadata (6.6 kB)
Collecting torchaudio
Downloading torchaudio-2.2.2-cp310-cp310-macosx_10_13_x86_64.whl.metadata (6.4 kB)
Requirement already satisfied: filelock in ./venv/lib/python3.10/site-packages (from torch) (3.16.1)
Requirement already satisfied: typing-extensions>=4.8.0 in ./venv/lib/python3.10/site-packages (from torch) (4.12.2)
Requirement already satisfied: sympy in ./venv/lib/python3.10/site-packages (from torch) (1.13.3)
Requirement already satisfied: networkx in ./venv/lib/python3.10/site-packages (from torch) (3.4.2)
Requirement already satisfied: jinja2 in ./venv/lib/python3.10/site-packages (from torch) (3.1.4)
Requirement already satisfied: fsspec in ./venv/lib/python3.10/site-packages (from torch) (2024.10.0)
Requirement already satisfied: numpy in ./venv/lib/python3.10/site-packages (from torchvision) (1.26.2)
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in ./venv/lib/python3.10/site-packages (from torchvision) (9.5.0)
Requirement already satisfied: MarkupSafe>=2.0 in ./venv/lib/python3.10/site-packages (from jinja2->torch) (2.1.5)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in ./venv/lib/python3.10/site-packages (from sympy->torch) (1.3.0)
Downloading torch-2.2.2-cp310-none-macosx_10_9_x86_64.whl (150.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 150.8/150.8 MB 26.1 MB/s eta 0:00:00
Downloading torchvision-0.17.2-cp310-cp310-macosx_10_13_x86_64.whl (1.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.7/1.7 MB 33.5 MB/s eta 0:00:00
Downloading torchaudio-2.2.2-cp310-cp310-macosx_10_13_x86_64.whl (3.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 34.3 MB/s eta 0:00:00
Installing collected packages: torch, torchvision, torchaudio
Successfully installed torch-2.2.2 torchaudio-2.2.2 torchvision-0.17.2

[hvaara]

Oh sorry, I just realized this was for an Intel Mac. I'm afraid bfloat16 isn't supported for that architecture.

[santi-mir]

Wasn't this solved with 2.6, i.e pytorch nightly in the selection matrix? I had a bfloat16 warning that disappeared, for ARM though. But worth trying. Make sure it's installed printing torch.__version__ if you try @SkyForceCoder

[SkyForceCoder]

@ghsanti the latest version is 2.2.2

% pip install torch==2.6.0
ERROR: Could not find a version that satisfies the requirement torch==2.6.0 (from versions: 1.11.0, 1.12.0, 1.12.1, 1.13.0, 1.13.1, 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1, 2.2.2)
ERROR: No matching distribution found for torch==2.6.0

% conda install pytorch=2.6.0 -c pytorch
Channels:

• pytorch
• intel
• conda-forge
• defaults
  Platform: osx-64
  Collecting package metadata (repodata.json): failed

UnavailableInvalidChannel: HTTP 403 FORBIDDEN for channel intel https://conda.anaconda.org/intel

The channel is not accessible or is invalid.

You will need to adjust your conda configuration to proceed.
Use conda config --show channels to view your configuration's current state,
and use conda config --show-sources to view config file locations.

[santi-mir]

You need to is the nightly url for mac, which is (remove vision and audio if not needed.)

pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cpu

I recommend you not to use conda anymore since they put more effort in pip iirc.

@SkyForceCoder

[SkyForceCoder]

@ghsanti
% pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cpu
Looking in indexes: https://download.pytorch.org/whl/nightly/cpu
ERROR: Could not find a version that satisfies the requirement torch (from versions: none)
ERROR: No matching distribution found for torch

[santi-mir]

You can click in the index and check which mac os they have. I guess your processor isnt ARM but x86_64 right?

I thought it was fixed for both architectures just using MPS. But, it may not be yet there.

For apple silicon arm64 that command runs just fine, assuming python 3.10+

@SkyForceCoder