run with "mps" is error:Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead.

[varhuman]

when I follow the automatic_mask_generator_example to generating masks, It works in my rtx3080 and m1pro's cpu, but when I change the device to 'mps',the error is show:"Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead."

this is my code:

    # 打开图片并转换为numpy数组
    image = cv2.imread(image_path)
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

    # get image_path name
    file_name = os.path.splitext(os.path.basename(image_path))[0]
    # 创建 output 文件夹（如果不存在）
    image_folder = os.path.join(OutPutFolder, file_name)
    check_floder(image_folder)

    # 获取当前时间戳
    start_time = time.time()

    # 根据输入图像生成多个区域掩码
    masks = mask_generator.generate(image)  ###### **error in here**

    # 获取当前时间戳并计算消耗时间
    end_time = time.time()
    elapsed_time = end_time - start_time

    print(f"Time elapsed for mask_generator.generate{file_name}: {elapsed_time:.2f} seconds")

    # 创建一个空白图像，用于存储融合的结果
    result_image = np.zeros_like(image, dtype=np.float32)

[varhuman]

this is more useful infomation:

sam_checkpoint = "./models/sam_vit_h_4b8939.pth"
device = 'cuda' if torch.cuda.is_available() else 'mps'
model_type = "default"

sam = sam_model_registry[model_type](checkpoint=sam_checkpoint)
sam.to(device=device) 
sam.to(dtype=torch.float32) # I try this but didn't work

and pytorch version is 2.0.0

[mambarek]

it works with device='cpu'

[varhuman]

it works with device='cpu'

I know, I've tried it before, but it's too slow I want to go faster :(

[ElpadoCan]

Same here. Since PyTorch officially supports Apple silicon GPU acceleration it would be nice if SAM would also do that.

[albert-ying]

Same here

[Bard2803]

The same here.

[ad12]

+1

[natwille1]

same

[ElpadoCan]

same here :(

Has anyone figured out a solution?
best regards

There is already a PR, but unfortunately the speed up is quite marginal compared to CUDA.

#122

[adamtheadmin]

A very insightful response from GPT-4:

This error message indicates that the Metal Performance Shaders (MPS) framework doesn't support float64 (double precision floating point numbers). Instead, you should use float32 (single precision floating point numbers).

Here's what you can do to fix this:

You need to convert equalized_depth_array to a float32 numpy array before converting it into a PyTorch tensor. You can do this by using the astype function in numpy:

python

equalized_depth_tensor = torch.from_numpy(equalized_depth_array.astype(np.float32)).to(depth_tensor.device)

This code will convert equalized_depth_array to a numpy array of type float32 before converting it to a PyTorch tensor. After that, it moves the tensor to the device specified by depth_tensor.device.

Note that using float32 instead of float64 will reduce the precision of the numbers in equalized_depth_array. This is usually not a problem in practice, especially in deep learning where float32 is commonly used, but it's something to be aware of.

[erychist]

A very insightful response from GPT-4:

This error message indicates that the Metal Performance Shaders (MPS) framework doesn't support float64 (double precision floating point numbers). Instead, you should use float32 (single precision floating point numbers).

Here's what you can do to fix this:

You need to convert equalized_depth_array to a float32 numpy array before converting it into a PyTorch tensor. You can do this by using the astype function in numpy:

python

equalized_depth_tensor = torch.from_numpy(equalized_depth_array.astype(np.float32)).to(depth_tensor.device)

This code will convert equalized_depth_array to a numpy array of type float32 before converting it to a PyTorch tensor. After that, it moves the tensor to the device specified by depth_tensor.device.

Note that using float32 instead of float64 will reduce the precision of the numbers in equalized_depth_array. This is usually not a problem in practice, especially in deep learning where float32 is commonly used, but it's something to be aware of.

Hello, and what should you do with this code? sorry i am pretty new in all these and i dont really understand ;[

[adamtheadmin]

A very insightful response from GPT-4:
This error message indicates that the Metal Performance Shaders (MPS) framework doesn't support float64 (double precision floating point numbers). Instead, you should use float32 (single precision floating point numbers).
Here's what you can do to fix this:
You need to convert equalized_depth_array to a float32 numpy array before converting it into a PyTorch tensor. You can do this by using the astype function in numpy:
python

equalized_depth_tensor = torch.from_numpy(equalized_depth_array.astype(np.float32)).to(depth_tensor.device)

This code will convert equalized_depth_array to a numpy array of type float32 before converting it to a PyTorch tensor. After that, it moves the tensor to the device specified by depth_tensor.device.
Note that using float32 instead of float64 will reduce the precision of the numbers in equalized_depth_array. This is usually not a problem in practice, especially in deep learning where float32 is commonly used, but it's something to be aware of.

Hello, and what should you do with this code? sorry i am pretty new in all these and i dont really understand ;[

Oh, I used my IDE to search for this:

equalized_depth_tensor =
And I replaced that line with

equalized_depth_tensor = torch.from_numpy(equalized_depth_array.astype(np.float32)).to(depth_tensor.device)

After that I had no more issues.

[ernstoller]

im debutant ... I use a Mac mini M2. I install stable diffusion with "Pinokio" . when I use deforum , I ve got this error message " Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn’t support float64. Please use float32 instead"... is anyone can help me ? do i have to edit a file? which ?

[philipkrueger]

I had the same issue running Auto1111 on M1 Max and struggled a while. Loading a different checkpoint solved it ¯_(ツ)_/¯

[ingvictoriano]

I had the same issue running Auto1111 on M1 Max and struggled a while. Loading a different checkpoint solved it ¯_(ツ)_/¯

Yes! This worked! Load a different checkpoint, then reload the one you want to use and it works. Thanks!

[WISON-Q]

我在 M1 Max 上运行 Auto1111 时遇到了同样的问题，并挣扎了一段时间。加载不同的检查点解决了问题 ́_(ツ)_/̵

very good,hahahah

[Anant5Gokhale]

I had the same issue running Auto1111 on M1 Max and struggled a while. Loading a different checkpoint solved it ¯_(ツ)_/¯

Are you referring to SAM checkpoint? I tried with other ViT-B and ViT-L checkpoints. The error still persists. How did you solve the problem?

[ram-ch]

I am using sam_vit_h_4b8939.pth file. Facing the same issue. Any solution to this issues?
Thanks in advance.

[VGabby]

replace this line:

segment-anything/segment_anything/automatic_mask_generator.py

Line 277 in 6fdee8f

in_points = torch.as_tensor(transformed_points, device=self.predictor.device)

with
in_points= torch.from_numpy(transformed_points.astype(np.float32)).to(self.predictor.device)
will make it work in mps

[malsaidi93]

replace this line:

segment-anything/segment_anything/automatic_mask_generator.py

Line 277 in 6fdee8f

in_points = torch.as_tensor(transformed_points, device=self.predictor.device)

with
in_points= torch.from_numpy(transformed_points.astype(np.float32)).to(self.predictor.device)
will make it work in mps

I casted transformed_points as type float32 without changing the method to torch.from_numpy and it worked
File https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/automatic_mask_generator.py line 277

in_points = torch.as_tensor(transformed_points.astype(np.float32), device=self.predictor.device)

[rfan-debug]

@malsaidi93

I got the same issue on MPS backends, too. I tried your approach but still got the same issue. Here are my environments:

torch                                 2.0.1
torchvision                           0.15.2

Any ideas?

I have already converted my tensor to have dtype as float32 and confirmed it in the notebook, but still got the same issue.

---

EDIT: never mind. I just noticed that i need to install your fork rather than the existing main (I thought that the commit had been merged. )

[adu-baffour]

self.double() is equivalent to self.to(torch.float64)
Use self.to(torch.float64) for mps devices.
https://pytorch.org/docs/stable/generated/torch.Tensor.double.html

[james397520]

if platform.system() == "Linux":
        device = "cuda"

elif platform.system() == "Darwin": #MacOS
        device = "mps"

[fabiotgolo]

im debutant ... I use a Mac mini M2. I install stable diffusion with "Pinokio" . when I use deforum , I ve got this error message " Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn’t support float64. Please use float32 instead"... is anyone can help me ? do i have to edit a file? which ?

Same problem here with Macbook M2

[winterdrive]

Replace this line:

segment-anything/segment_anything/automatic_mask_generator.py

Line 277 in 6fdee8f

in_points = torch.as_tensor(transformed_points, device=self.predictor.device)

with
in_points = torch.from_numpy(transformed_points.astype(np.float32)).to(self.predictor.device)
to make it work in MPS.

I cast transformed_points as type float32 without changing the method to torch.from_numpy, and it worked. File: automatic_mask_generator.py line 277.

in_points = torch.as_tensor(transformed_points.astype(np.float32), device=self.predictor.device)

I tried both methods, but it seems there's a memory leak issue when processing multiple images. Did you encounter the same issue after revising the code in automatic_mask_generator.py?

Terminal output (I have five images, but it stops at the third one):

"/Volumes/Seagate Bac/materials_maker/.venv/bin/python" /Volumes/Seagate Bac/materials_maker/segment_anything_poc.py 
Processed 417544405_3674148672863770_9184643183196172667_n.jpg in 70.24 seconds
Processed 429654242_8180557191961396_1072716505924690909_n.jpg in 55.64 seconds
Processed 429974052_8180557028628079_175881528888267460_n.jpg in 137.48 seconds

Process finished with exit code 133 (interrupted by signal 5:SIGTRAP)

My code:

import os
import cv2
import supervision as sv
import torch
from segment_anything import sam_model_registry, SamAutomaticMaskGenerator
import time


def download_model_weights(model_weights, download_link):
    """
    Download model weights if not already downloaded.

    Args:
        model_weights (str): Path to the model weights file.
        download_link (str): URL to download the model weights from.
    """
    if not os.path.exists(model_weights):
        # Download the weights file
        os.system(f"curl {download_link} -o {model_weights}")


def load_model(model_type, runtime_env="local"):
    """
    Load the SAM model based on the provided type.

    Args:
        model_type (str): Type of SAM model to load.
        runtime_env (str): Runtime environment (default is "local").

    Returns:
        torch.nn.Module: Loaded SAM model.
    """
    # Define paths to SAM checkpoints
    checkpoints = {
        "vit_b": "sam/sam_vit_b_01ec64.pth",
        "vit_l": "sam/sam_vit_l_0b3195.pth",
        "vit_h": "sam/sam_vit_h_4b8939.pth"
    }

    model_weights = checkpoints[model_type]
    download_links = {
        "vit_b": "https://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pth",
        "vit_l": "https://dl.fbaipublicfiles.com/segment_anything/sam_vit_l_0b3195.pth",
        "vit_h": "https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth"
    }

    download_model_weights(model_weights, download_links[model_type])

    # Set device

    ## Seems to have the memory leak issue
    if torch.backends.mps.is_available():
        device = "mps"
    else:
        device = "cuda" if torch.cuda.is_available() else "cpu"

    # Load SAM model
    SAM = sam_model_registry[model_type](checkpoint=model_weights).to(device=device)
    return SAM


def process_images_in_directory(image_dir, output_dir, model_type="vit_h", runtime_env="local"):
    """
    Process images in a directory using the SAM model.

    Args:
        image_dir (str): Directory containing input images.
        output_dir (str): Directory to save annotated images.
        model_type (str): Type of SAM model to use (default is "vit_h").
        runtime_env (str): Runtime environment (default is "local").
    """
    # Load SAM model
    SAM = load_model(model_type, runtime_env)

    # Create output directory if it doesn't exist
    os.makedirs(output_dir, exist_ok=True)

    # Process each image in the directory
    for filename in os.listdir(image_dir):
        import numpy as np
        if filename.lower().endswith((".jpg", ".jpeg", ".png")):
            # Record start time
            start_time = time.time()

            # Read the image
            image_path = os.path.join(image_dir, filename)
            with open(image_path, "rb") as f:
                image_bytes = f.read()
            image_np = cv2.imdecode(np.frombuffer(image_bytes, np.uint8), cv2.IMREAD_COLOR)
            image_rgb = cv2.cvtColor(image_np, cv2.COLOR_BGR2RGB)

            # Generate segmentation masks
            sam_result = SamAutomaticMaskGenerator(SAM).generate(image_rgb)

            # Convert SAM result to supervision format
            detections = sv.Detections.from_sam(sam_result=sam_result)

            # Annotate the image with masks
            mask_annotator = sv.MaskAnnotator(color_lookup=sv.ColorLookup.INDEX)
            annotated_image = mask_annotator.annotate(scene=image_np.copy(), detections=detections)

            # Save the annotated image
            output_path = os.path.join(output_dir, filename)
            cv2.imwrite(output_path, annotated_image[:, :, ::-1])  # Save as BGR format

            elapsed_time = time.time() - start_time
            print(f"Processed {filename} in {elapsed_time:.2f} seconds")


def main():
    image_dir = "/Volumes/Seagate Bac/materials_maker/fern/input/"
    output_dir = "/Volumes/Seagate Bac/materials_maker/fern/output/"
    process_images_in_directory(image_dir, output_dir, model_type="vit_h", runtime_env="local")


if __name__ == "__main__":
    main()