black canvas on safari for MediaPipe Selfie Segmentation in javascript

[nicolasoros]

Hello,

I'm having issues rendering the output for the MediaPipe Selfie Segmentation on Safari only.

The canvas element that should display the video with the overlay is all black.

I see the same problem with the official demo examples from https://codepen.io/mediapipe/pen/wvJyQpq, and with my own code.

It seems that both results.image and results.segmentationMask used in the function onResults(results: mpSelfieSegmentation.Results) are ImageBitmap.

But it seems that ImageBitmap is not supported on Safari: https://developer.mozilla.org/en-US/docs/Web/API/ImageBitmap
Could that be the issue?

Safari version: 15.0

Thanks

[tyrmullen]

That's quite possible, but in any case I suspect this is a Safari 15-only issue, since Safari 15 now has support for WebGL2, so if you try with a different version of Safari I imagine you should be fine. Of course, we'd like to support Safari 15 as well, so having an issue tracking that seems like a good idea if that is indeed the problem here.

[nicolasoros]

thanks.

The demo doesn't crash for me. And I can see that the model is being updated since the function onResults() keeps being called for each frame.

Platform:

• MacOS BigSur 11.6
• Macbook Air M1 2020
• Safari Version 15.0 (16612.1.29.41.4, 16612)

Log:
[Warning] I1015 15:29:20.482000 1 gl_context_webgl.cc:151] Successfully created a WebGL context with major version 3 and handle 3 (selfie_segmentation_solution_wasm_bin.js, line 9)
[Warning] I1015 15:29:20.484000 1 gl_context.cc:349] GL version: 3.0 (OpenGL ES 3.0 (WebGL 2.0)) (selfie_segmentation_solution_wasm_bin.js, line 9)
[Warning] W1015 15:29:20.484000 1 gl_context.cc:819] OpenGL error checking is disabled (selfie_segmentation_solution_wasm_bin.js, line 9)

In the meantime, is there an API documentation for the onResults() function?
I'd like to see if I can get the raw data of the mask as a uint8 array, or even the output model prediction, instead of the results.segmentationMask ImageBitmap.

[tyrmullen]

I still suspect there's a good chance the issues are related. Note that the WebGL context has major version 3 (Meaning ES 3.0, which corresponds to WebGL2).

Unfortunately the only documentation aside from the basic overview on mediapipe.dev or the CodePen demos would be the TypeScript definitions themselves (*.d.ts), but those are definitely the most informative and can be browsed directly on jsdelivr (example for pose: https://cdn.jsdelivr.net/npm/@mediapipe/pose@0.4.1633558788/index.d.ts).

[google-ml-butler]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[google-ml-butler]

Closing as stale. Please reopen if you'd like to work on this further.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No

[nicolasoros]

Hi @tyrmullen and @sgowroji ,
has this problem been investigated or fixed?
Should we re-open this issue?

[tyrmullen]

The problem has not been investigated, and I don't have an ETA for doing so at this time. Hopefully things will get better from the WebKit side in the meantime?

[ayushgdev]

Hello @nicolasoros
We are revisiting older issues. This issue is not reproducible in Safari. The web demo as well as code in documentation works fine at 60 fps in

• Macbook Pro M1,
• McOS 13.2.1
• Safari Version 16.3 (18614.4.6.1.6)

For the Javascript API of onResults(), we have released the Typescript type definition #1408 (comment) which should give an idea on the possible options and APIs. For full source code release, please check the comment #1408 (comment).

[google-ml-butler]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[google-ml-butler]

Closing as stale. Please reopen if you'd like to work on this further.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No