selfie_multiclass task with option delegate=BaseOptions.Delegate.GPU not working

[SoaringTiger]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

None

OS Platform and Distribution

Ubuntu 22.04.3 LTS (docker image: tensorflow:2.16.1-gpu-jupyter)

MediaPipe Tasks SDK version

0.10.14

Task name (e.g. Image classification, Gesture recognition etc.)

selfie_multiclass

Programming Language and version (e.g. C++, Python, Java)

Python

Describe the actual behavior

delegate=BaseOptions.Delegate.GPU not work

Describe the expected behaviour

enable GPU support

Standalone code/steps you may have used to try to get what you need

import mediapipe as mp
import numpy as np
import cv2

model_path = '/models/mediapipe/selfie_multiclass_256x256.tflite'

IMAGE_FILE = 'input.jpg'

BaseOptions = mp.tasks.BaseOptions
ImageSegmenter = mp.tasks.vision.ImageSegmenter
ImageSegmenterOptions = mp.tasks.vision.ImageSegmenterOptions
VisionRunningMode = mp.tasks.vision.RunningMode


# ImageSegmenter 
options = ImageSegmenterOptions(
    #base_options=BaseOptions(model_asset_path=model_path),
    base_options = BaseOptions(model_asset_path=model_path, delegate=BaseOptions.Delegate.GPU),
    running_mode=VisionRunningMode.IMAGE,
    output_category_mask=True)

colors = [
    [255, 0, 0],  # background: blue
    [255, 255, 0],  # hair: skyblue
    [0, 255, 0],  # body-skin: green
    [255, 0, 128],  # face-skin: purple
    [255, 0, 255],  # clothes: magenta
    [0, 128, 255]  # others: orange
]

category = ["background", "hair", "body-skin", "face-skin", "clothes", "others"]


with ImageSegmenter.create_from_options(options) as segmenter:

    mp_image = mp.Image.create_from_file(IMAGE_FILE)

    original_image = mp_image.numpy_view()
    # original_image = original_image[:, :, :3]  # remove alpha channel

    # masks
    segmented_masks = segmenter.segment(mp_image)

    # category_mask
    category_mask = segmented_masks.category_mask
    category_mask_np = category_mask.numpy_view()

    h, w = category_mask_np.shape # numpy  width, height 
    color_image = np.zeros((h, w, 3), dtype=np.uint8) #generate same size empty image
    for i, color in enumerate(colors): # fill every class
        color_image[category_mask_np == i] = color

    alpha = 0.5
    blended_image = cv2.addWeighted(original_image, 1 - alpha, color_image, alpha, 0)

    status = cv2.imwrite('path_to_save_image.jpg', blended_image)
    print("Image written to file-system: ", status)

Other info / Complete Logs

When I set base_options=BaseOptions(model_asset_path=model_path), the code worked, when I set base_options = BaseOptions(model_asset_path=model_path, delegate=BaseOptions.Delegate.GPU), the code got error:
'''
2024-06-25 02:26:50.945231: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1719282412.338260     551 task_runner.cc:85] GPU suport is not available: INTERNAL: ; RET_CHECK failure (mediapipe/gpu/gl_context_egl.cc:77) display != EGL_NO_DISPLAYeglGetDisplay() returned error 0x300c
Traceback (most recent call last):
  File "/tools/mask_1_img.py", line 37, in <module>
    with ImageSegmenter.create_from_options(options) as segmenter:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/mediapipe/tasks/python/vision/image_segmenter.py", line 268, in create_from_options
    return cls(
           ^^^^
  File "/usr/local/lib/python3.11/dist-packages/mediapipe/tasks/python/vision/image_segmenter.py", line 145, in __init__
    super(ImageSegmenter, self).__init__(
  File "/usr/local/lib/python3.11/dist-packages/mediapipe/tasks/python/vision/core/base_vision_task_api.py", line 70, in __init__
    self._runner = _TaskRunner.create(graph_config, packet_callback)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Service "kGpuService", required by node mediapipe_tasks_vision_image_segmenter_imagesegmentergraph__mediapipe_tasks_core_inferencesubgraph__inferencecalculator__mediapipe_tasks_vision_image_segmenter_imagesegmentergraph__mediapipe_tasks_core_inferencesubgraph__InferenceCalculator, was not provided and cannot be created: ; RET_CHECK failure (mediapipe/gpu/gl_context_egl.cc:77) display != EGL_NO_DISPLAYeglGetDisplay() returned error 0x300c
'''

run the script in tensorflow:2.16.1-gpu-jupyter container
import tensorflow as tf
print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))

'''
>>> import tensorflow as tf
2024-06-25 03:53:58.671721: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
>>> print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))
2024-06-25 03:54:00.062634: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-06-25 03:54:00.067000: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-06-25 03:54:00.067152: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
Num GPUs Available:  1
'''

[kuaashish]

Hi @SoaringTiger,

Thank you for bringing this to our attention again. It appears that our multi-class selfie segmentation currently does not support GPU usage. A similar issue was reported previously at #5081. Please allow us some time to investigate further, and I will provide an update soon.

Thank you!!

[kuaashish]

Hi @SoaringTiger,

Could you confirm if you can run other examples in your GPU environment, such as the Hand example provided here?

Thank you!!

[SoaringTiger]

Hi @SoaringTiger,

Could you confirm if you can run other examples in your GPU environment, such as the Hand example provided here?

Thank you!!

Hi @kuaashish

# STEP 2: Create an HandLandmarker object.
base_options = python.BaseOptions(
    model_asset_path='/models/mediapipe/hand_landmarker.task', 
    #delegate=BaseOptions.Delegate.GPU
    )
options = vision.HandLandmarkerOptions(base_options=base_options,
                                       num_hands=2)
detector = vision.HandLandmarker.create_from_options(options)

When I run the code without GPU, got warning, but I can get result image:

I0000 00:00:1720067300.162163     406 task_runner.cc:85] GPU suport is not available: INTERNAL: ; RET_CHECK failure (mediapipe[/gpu/gl_context_egl.cc:77](http://192.168.2.8:8888/gpu/gl_context_egl.cc#line=76)) display != EGL_NO_DISPLAYeglGetDisplay() returned error 0x300c
W0000 00:00:1720067300.170902     948 inference_feedback_manager.cc:114] Feedback manager requires a model with a single signature inference. Disabling support for feedback tensors.
W0000 00:00:1720067300.186529     948 inference_feedback_manager.cc:114] Feedback manager requires a model with a single signature inference. Disabling support for feedback tensors.

When I run the code with 'delegate=BaseOptions.Delegate.GPU', I got error:

WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1720067775.707569    1064 task_runner.cc:85] GPU suport is not available: INTERNAL: ; RET_CHECK failure (mediapipe/gpu/gl_context_egl.cc:77) display != EGL_NO_DISPLAYeglGetDisplay() returned error 0x300c
Traceback (most recent call last):
  File "/sam/mp_mask_1_hands.py", line 65, in <module>
    detector = vision.HandLandmarker.create_from_options(options)
  File "/usr/local/lib/python3.10/dist-packages/mediapipe/tasks/python/vision/hand_landmarker.py", line 364, in create_from_options
    return cls(
  File "/usr/local/lib/python3.10/dist-packages/mediapipe/tasks/python/vision/core/base_vision_task_api.py", line 70, in __init__
    self._runner = _TaskRunner.create(graph_config, packet_callback)
RuntimeError: Service "kGpuService", required by node mediapipe_tasks_vision_hand_landmarker_handlandmarkergraph__mediapipe_tasks_vision_hand_detector_handdetectorgraph__mediapipe_tasks_core_inferencesubgraph__inferencecalculator__mediapipe_tasks_vision_hand_landmarker_handlandmarkergraph__mediapipe_tasks_vision_hand_detector_handdetectorgraph__mediapipe_tasks_core_inferencesubgraph__InferenceCalculator, was not provided and cannot be created: ; RET_CHECK failure (mediapipe/gpu/gl_context_egl.cc:77) display != EGL_NO_DISPLAYeglGetDisplay() returned error 0x300c

[kuaashish]

Hi @SoaringTiger,

It seems that some drivers are missing, preventing the GPU from being detected in the environment. Can you please try this Colab for benchmarking some of the tasks: Colab Notebook? Additionally, could you adopt the same for your environment and update us on the status?

Thank you!!

[SoaringTiger]

Hi @SoaringTiger,

It seems that some drivers are missing, preventing the GPU from being detected in the environment. Can you please try this Colab for benchmarking some of the tasks: Colab Notebook? Additionally, could you adopt the same for your environment and update us on the status?

Thank you!!

Hi, @kuaashish
my nvidia-smi info in docker container, docker image is tensorflow/tensorflow:2.16.2-gpu-jupyter
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.29.06 Driver Version: 545.29.06 CUDA Version: 12.3 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 4060 Ti Off | 00000000:01:00.0 On | N/A |
| 0% 52C P8 8W / 165W | 14558MiB / 16380MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

I have install the mediapipe-nightly and I can only run mediapipe task script without gpu option: colab

I have tested tensorflow with gpu in container

import tensorflow as tf

print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))

matrix1 = tf.random.uniform([1000, 1000])
matrix2 = tf.random.uniform([1000, 1000])

with tf.device('/GPU:0'):
    product = tf.matmul(matrix1, matrix2)

with tf.device('/GPU:0'):
    print("Matrix multiplication result on GPU:")
    print(product)

Got result

Num GPUs Available:  1
Matrix multiplication result on GPU:
tf.Tensor(
[[243.63484 238.3081  250.76256 ... 249.20293 241.33276 251.67079]
 [249.44817 234.74933 245.9238  ... 242.76912 239.75523 247.66855]
 [238.73503 231.70839 242.5265  ... 243.06746 235.40015 246.4232 ]
 ...
 [236.47717 229.38445 241.08224 ... 236.93318 233.91486 245.26237]
 [246.1244  242.03398 246.94525 ... 250.89856 238.61884 247.93216]
 [248.92546 240.55031 253.50462 ... 248.77428 250.05797 252.9958 ]], shape=(1000, 1000), dtype=float32)