Got it working on MacBook (MPS)

[Grant-CP]

I have a version of these nodes working via MPS for those with macbooks. On my M1 Pro 32GB it took 60 seconds for 32 frames and 650 seconds for 600 frames. So about 1 second per animation frame.

Repo is here. I welcome all pull requests from other macbook users.
https://github.com/Grant-CP/ComfyUI-LivePortraitKJ-MPS

For @kijai I'm not sure that these changes can be added into the main as there are a few places where I changed numerical precision. So I would expect my version to have slightly worse performance. I am more than happy to talk about my changes if you want to start supporting MPS as a back end when you make nodes. Sorry for not forking correctly. I'm still learning how to use git.

[tryx78]

for me now work thanks with @Grant-CP repo
thanks
i use M3 Pro 14inch - 18gb ram

[Grant-CP]

@tryx78 Glad it's working for you! Can you tell me how long it takes? Just the overall time for prompt evaluation and how many frames you set in the video import node.

Also, can you confirm that it runs just fine without PYTORCH_ENABLE_MPS_FALLBACK=1 python main.py to set the proper fallback for comfyui? I believe that was necessary on my M1, but I forgot to write it in the repo originally.
@melMass I'm not sure if setting an environment variable like this fits into your patch solution.

@melMass Thanks. Good idea with the patch. A change I didn't make is that the code is still using CUDA as the execution provider for onnx networks in that file. I think it will just fallback to CPU, but it would be nice to not have the error message any more. I know in comfyui there is also the CoreML execution provider which I believe is fairly new but would be great to use if it works.

[tryx78]

@Grant-CP now i have this error
Error occurred when executing LivePortraitProcess: The operator 'aten::grid_sampler_3d' is not currently implemented for the MPS device. If you want this op to be added in priority during the prototype phase of this feature, please comment on pytorch/pytorch#77764. As a temporary fix, you can set the environment variable PYTORCH_ENABLE_MPS_FALLBACK=1 to use the CPU as a fallback for this op. WARNING: this will be slower than running natively on MPS.

how fix this?

[tryx78]

@Grant-CP
48 fps in 38.80 seconds with PYTORCH_ENABLE_MPS_FALLBACK=1
48 fsp in 53.46 seconds whitout PYTORCH_ENABLE_MPS_FALLBACK=1

[Grant-CP]

@tryx78 Thanks so much! I added the Pytorch fallback to the README of my repo.

Good to see that your M3 is so much faster. Let me know if you run into any other issues!

[kijai]

I changed all the hardcoded cuda stuff to use the comfy detection, and put that one tensor operation in try/except block as it probably fails on MPS, but I can't test if that's enough.

[Grant-CP]

@kijai Great I've looked through the changes and I will test them out later.

In line 62 of liveportrait/modules/dense_motion.py, if you want to change the code for the assertion error I believe the error is: AttributeError: module 'torch.mps' has no attribute 'FloatTensor'.

Its a stupid error that's been in PyTorch for multiple years. Presumably the error will change from AttributeError to something more reasonable in a future version of pytorch.

[Grant-CP]

@kijai I had a chance to test it and I get a torch autocast error. This might be a bug on comfy's end as it seems to me like their get_autocast_device should handle mps not being supported. See the error here:

Traceback (most recent call last):
  File "/Users/grant/Documents/Repos/ComfyUI/execution.py", line 151, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/grant/Documents/Repos/ComfyUI/execution.py", line 81, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/grant/Documents/Repos/ComfyUI/execution.py", line 74, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/grant/Documents/Repos/ComfyUI/custom_nodes/ComfyUI-LivePortraitKJ/nodes.py", line 283, in process
    cropped_frames, full_frame = pipeline.execute(img, driving_images_np)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/grant/Documents/Repos/ComfyUI/custom_nodes/ComfyUI-LivePortraitKJ/liveportrait/live_portrait_pipeline.py", line 53, in execute
    x_s_info = self.live_portrait_wrapper.get_kp_info(I_s)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/grant/Documents/Repos/ComfyUI/custom_nodes/ComfyUI-LivePortraitKJ/liveportrait/live_portrait_wrapper.py", line 95, in get_kp_info
    with torch.autocast(device_type=get_autocast_device(self.device_id), dtype=torch.float16, enabled=self.cfg.flag_use_half_precision):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/comfyui/lib/python3.11/site-packages/torch/amp/autocast_mode.py", line 241, in __init__
    raise RuntimeError(
RuntimeError: User specified an unsupported autocast device_type 'mps'

In my repo I was only able to get it working by disabling autocast entirely (not even setting device to cpu works). For example:

with torch.no_grad():
            #HACK
            # with torch.autocast(device_type='cpu', dtype=torch.float16, enabled=self.cfg.flag_use_half_precision):
            #     feature_3d = self.appearance_feature_extractor(x)
            feature_3d = self.appearance_feature_extractor(x)

Again, I'm not sure if there's a way to fix this with comfy's autocast manager. I won't have time to look into that today.

[kijai]

@kijai I had a chance to test it and I get a torch autocast error. This might be a bug on comfy's end as it seems to me like their get_autocast_device should handle mps not being supported. See the error here:

Traceback (most recent call last):
  File "/Users/grant/Documents/Repos/ComfyUI/execution.py", line 151, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/grant/Documents/Repos/ComfyUI/execution.py", line 81, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/grant/Documents/Repos/ComfyUI/execution.py", line 74, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/grant/Documents/Repos/ComfyUI/custom_nodes/ComfyUI-LivePortraitKJ/nodes.py", line 283, in process
    cropped_frames, full_frame = pipeline.execute(img, driving_images_np)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/grant/Documents/Repos/ComfyUI/custom_nodes/ComfyUI-LivePortraitKJ/liveportrait/live_portrait_pipeline.py", line 53, in execute
    x_s_info = self.live_portrait_wrapper.get_kp_info(I_s)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/grant/Documents/Repos/ComfyUI/custom_nodes/ComfyUI-LivePortraitKJ/liveportrait/live_portrait_wrapper.py", line 95, in get_kp_info
    with torch.autocast(device_type=get_autocast_device(self.device_id), dtype=torch.float16, enabled=self.cfg.flag_use_half_precision):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/comfyui/lib/python3.11/site-packages/torch/amp/autocast_mode.py", line 241, in __init__
    raise RuntimeError(
RuntimeError: User specified an unsupported autocast device_type 'mps'

In my repo I was only able to get it working by disabling autocast entirely (not even setting device to cpu works). For example:

with torch.no_grad():
            #HACK
            # with torch.autocast(device_type='cpu', dtype=torch.float16, enabled=self.cfg.flag_use_half_precision):
            #     feature_3d = self.appearance_feature_extractor(x)
            feature_3d = self.appearance_feature_extractor(x)

Again, I'm not sure if there's a way to fix this with comfy's autocast manager. I won't have time to look into that today.

I've done it like that before too yeah, can just the manager mps detection to skip the whole autocast conditionally.

[kijai]

@Grant-CP Can you try now? Skipping the whole autocast based on the dtype.

[Grant-CP]

@kijai I'm getting the same error. Truncated error is below. I don't believe I've set any half_precision flags so I assume that comes from elsewhere in the code. I am using the fp16 models though, as I assume most people would.

do you think try:... except RuntimeError: would be bad? Another option would be to use torch.backends.mps.is_available(). That should return true only on macbooks. Not sure if there's another issue you are trying to solve with moving the half_precision flag out of the autocast() call though.

File "/Users/grant/Documents/Repos/ComfyUI/custom_nodes/ComfyUI-LivePortraitKJ/liveportrait/live_portrait_pipeline.py", line 53, in execute
    x_s_info = self.live_portrait_wrapper.get_kp_info(I_s)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/grant/Documents/Repos/ComfyUI/custom_nodes/ComfyUI-LivePortraitKJ/liveportrait/live_portrait_wrapper.py", line 94, in get_kp_info
    with torch.autocast(get_autocast_device(self.device_id), dtype=torch.float16) if self.cfg.flag_use_half_precision else nullcontext():
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/comfyui/lib/python3.11/site-packages/torch/amp/autocast_mode.py", line 241, in __init__
    raise RuntimeError(
RuntimeError: User specified an unsupported autocast device_type 'mps'

[kijai]

@kijai I'm getting the same error. Truncated error is below. I don't believe I've set any half_precision flags so I assume that comes from elsewhere in the code. I am using the fp16 models though, as I assume most people would.

do you think try:... except RuntimeError: would be bad? Another option would be to use torch.backends.mps.is_available(). That should return true only on macbooks. Not sure if there's another issue you are trying to solve with moving the half_precision flag out of the autocast() call though.

File "/Users/grant/Documents/Repos/ComfyUI/custom_nodes/ComfyUI-LivePortraitKJ/liveportrait/live_portrait_pipeline.py", line 53, in execute
    x_s_info = self.live_portrait_wrapper.get_kp_info(I_s)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/grant/Documents/Repos/ComfyUI/custom_nodes/ComfyUI-LivePortraitKJ/liveportrait/live_portrait_wrapper.py", line 94, in get_kp_info
    with torch.autocast(get_autocast_device(self.device_id), dtype=torch.float16) if self.cfg.flag_use_half_precision else nullcontext():
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/comfyui/lib/python3.11/site-packages/torch/amp/autocast_mode.py", line 241, in __init__
    raise RuntimeError(
RuntimeError: User specified an unsupported autocast device_type 'mps'

This is with fp32 selected though? I should probably automate that.

[Grant-CP]

@kijai my apologies. I had the model loader pipeline set to fp16. So the idea of this code is to force MPS users to use fp32 for these particular models because MPS doesn't support autocast. My mistake! I think some other parts of comfyui will print a message saying "mixed precision is not supported on this device, reverting to full precision" or something like that. If you wanted to be nice to mac users you could do a check of cfg.flag_use_half_precision and if it's true then throw a descriptive error message.

It will be interesting to test whether setting it to fp32 will hurt performance. I'm not sure what the default was in the original repo, but I assume it was fp16 since I was also running into an error on this line originally?

Anyways, we are down to the last error, which is:

ile "/Users/grant/Documents/Repos/ComfyUI/custom_nodes/ComfyUI-LivePortraitKJ/liveportrait/modules/dense_motion.py", line 81, in forward
    deformed_feature = self.create_deformed_feature(feature, sparse_motion)  # (bs, 1+num_kp, c=4, d=16, h=64, w=64)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/grant/Documents/Repos/ComfyUI/custom_nodes/ComfyUI-LivePortraitKJ/liveportrait/modules/dense_motion.py", line 50, in create_deformed_feature
    sparse_deformed = F.grid_sample(feature_repeat, sparse_motions, align_corners=False)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/comfyui/lib/python3.11/site-packages/torch/nn/functional.py", line 4353, in grid_sample
    return torch.grid_sampler(input, grid, mode_enum, padding_mode_enum, align_corners)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
NotImplementedError: The operator 'aten::grid_sampler_3d' is not currently implemented for the MPS device. If you want this op to be added in priority during the prototype phase of this feature, please comment on https://github.com/pytorch/pytorch/issues/77764. As a temporary fix, you can set the environment variable `PYTORCH_ENABLE_MPS_FALLBACK=1` to use the CPU as a fallback for this op. WARNING: this will be slower than running natively on MPS.

I solved this one by running my comfyui with PYTORCH_ENABLE_MPS_FALLBACK=1 python main.py. One user near the top of this thread sounded like they maybe got it to work more slowly without this flag on their M3 macbook? They weren't totally clear.

An option might be to move both feature_repeat and sparse_motion to cpu before this operation? I assume that's what the fallback flag does. Unfortunately the pytorch function grid_sample looks like just a wrapper for non-python code.

[Grant-CP]

@kijai I can confirm that the node works with the env flag set. So your node is just as functional. I do think it is marginally slower, maybe because of the fp32 setting on the model loader? For example 57 seconds originally vs 60 seconds now for 32 frames.

I think it would be good to skip the autocast context on MPS, rather then relying on the MPS users loading everything into fp32. I'm going to check and see if I can skip the fallback call though