GPU Support for Pose Estimation on MacOS or Linux server

[Sun-Yize]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

Yes

OS Platform and Distribution

MacOS M2 Chip

MediaPipe Tasks SDK version

0.10.14

Task name (e.g. Image classification, Gesture recognition etc.)

Pose Estimation

Programming Language and version (e.g. C++, Python, Java)

Python

Describe the actual behavior

I tried to run the GPU version of mediapipe in macos, but I kept getting the following error. If it is set to CPU, that is, delegate=BaseOptions.Delegate.CPU, everything works fine

Describe the expected behaviour

Bug Fix

Standalone code/steps you may have used to try to get what you need

import mediapipe as mp
import cv2


BaseOptions = mp.tasks.BaseOptions
PoseLandmarker = mp.tasks.vision.PoseLandmarker
PoseLandmarkerOptions = mp.tasks.vision.PoseLandmarkerOptions
VisionRunningMode = mp.tasks.vision.RunningMode

options = PoseLandmarkerOptions(
    base_options=BaseOptions(
        model_asset_path='data/pose_landmarker_heavy.task',
        delegate=BaseOptions.Delegate.GPU
    ),
    running_mode=VisionRunningMode.IMAGE, 
)

mp_pose = mp.solutions.pose
mp_drawing = mp.solutions.drawing_utils

with PoseLandmarker.create_from_options(options) as landmarker:
    cap = cv2.VideoCapture('test2.mp4')
    fps = cap.get(cv2.CAP_PROP_FPS)
    frame_interval = int(fps // 5)
    count = 0
    while cap.isOpened():
        ret, image = cap.read()
        mp_image = mp.Image(image_format=mp.ImageFormat.SRGB, data=image)
        results = landmarker.detect(mp_image)
        count += 1
    cap.release()
    cv2.destroyAllWindows()

Other info / Complete Logs

WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1723543300.469173 1531245 gl_context.cc:357] GL version: 2.1 (2.1 Metal - 88.1), renderer: Apple M2
INFO: Created TensorFlow Lite delegate for Metal.
F0000 00:00:1723543300.834500 1531448 gpu_buffer_storage_cv_pixel_buffer.cc:154] Check failed: status_or_buffer is OK (UNKNOWN: ; unsupported ImageFrame format: 1) 
*** Check failure stack trace: ***
    @        0x12183c938  absl::log_internal::LogMessage::SendToLog()
    @        0x12183c380  absl::log_internal::LogMessage::Flush()
    @        0x12183cc64  absl::log_internal::LogMessageFatal::~LogMessageFatal()
    @        0x12183cc8c  absl::log_internal::LogMessageFatal::~LogMessageFatal()
    @        0x121210de4  mediapipe::ConvertFromImageFrame()
    @        0x121211654  std::__1::__invoke_void_return_wrapper<>::__call<>()
    @        0x12120f634  std::__1::invoke<>()
    @        0x12120e3d8  mediapipe::GpuBuffer::StorageHolder::GetStorageForView()
    @        0x12120ee58  mediapipe::GpuBuffer::GetStorageForViewOrDie()
    @        0x121193e28  mediapipe::Image::ConvertToGpu()
    @        0x120c77840  std::__1::__function::__func<>::operator()()
    @        0x1212154e8  mediapipe::GlContext::SwitchContextAndRun()
    @        0x121214c2c  mediapipe::GlContext::Run()
    @        0x120c8238c  mediapipe::GlCalculatorHelper::RunInGlContext()
    @        0x120c82550  mediapipe::GlCalculatorHelper::RunInGlContext()
    @        0x120c77558  mediapipe::GlCalculatorHelper::RunInGlContext<>()
    @        0x120c76ba4  mediapipe::api2::ImageCloneCalculator::Process()
    @        0x121226674  mediapipe::CalculatorNode::ProcessNode()
    @        0x121204ddc  mediapipe::internal::SchedulerQueue::RunCalculatorNode()
    @        0x121204878  mediapipe::internal::SchedulerQueue::RunNextTask()
    @        0x12121e9d8  mediapipe::ThreadPool::RunWorker()
    @        0x12121e3f4  mediapipe::ThreadPool::WorkerThread::ThreadBody()
    @        0x19014ef94  _pthread_start
    @        0x190149d34  thread_start
zsh: abort      python test_mediapipe.py

[Sun-Yize]

I also tried to run the same code in Google Colab, but got a different error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-16-40a8d7c37d16> in <cell line: 21>()
     19 mp_drawing = mp.solutions.drawing_utils
     20 
---> 21 with PoseLandmarker.create_from_options(options) as landmarker:
     22     cap = cv2.VideoCapture('/content/gdrive/MyDrive/data/pose/test1.mp4')
     23     fps = cap.get(cv2.CAP_PROP_FPS)

1 frames
/usr/local/lib/python3.10/dist-packages/mediapipe/tasks/python/vision/core/base_vision_task_api.py in __init__(self, graph_config, running_mode, packet_callback)
     68           'callback should not be provided.'
     69       )
---> 70     self._runner = _TaskRunner.create(graph_config, packet_callback)
     71     self._running_mode = running_mode
     72 

RuntimeError: Service "kGpuService", required by node mediapipe_tasks_vision_pose_landmarker_poselandmarkergraph__mediapipe_tasks_vision_pose_detector_posedetectorgraph__mediapipe_tasks_core_inferencesubgraph__inferencecalculator__mediapipe_tasks_vision_pose_landmarker_poselandmarkergraph__mediapipe_tasks_vision_pose_detector_posedetectorgraph__mediapipe_tasks_core_inferencesubgraph__InferenceCalculator, was not provided and cannot be created: ; RET_CHECK failure (mediapipe/gpu/gl_context_egl.cc:84) egl_initializedUnable to initialize EGL

Even installing the following dependencies does not resolve:

sudo apt-get install mesa-common-dev libegl1-mesa-dev libgles2-mesa-dev
sudo apt-get install mesa-utils

[kuaashish]

Hi @Sun-Yize,

An error is occurring because the image format is unsupported; our Metal implementation only supports an alpha channel. For further clarity, please review this specific 5265(comment)and let us know if you're now able to work in GPU mode.

Thank you!!

[Sun-Yize]

Hi @kuaashish
The answer solved my problem of not being able to run GPU on MacOS.

In addition, the issue mentioned above that cannot be run in Google Colab or Linux server has also been resolved. This suggestion #5505 helped me.

Thanks a lot!!

[kuaashish]

Hi @Sun-Yize,

Thank you for the confirmation. the issue has been resolved on your end, can we close the status of the issue and mark it resolved?