Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mediapipe CodePens don't run on iOS Safari #1427

Closed
ionif opened this issue Dec 23, 2020 · 59 comments
Closed

Mediapipe CodePens don't run on iOS Safari #1427

ionif opened this issue Dec 23, 2020 · 59 comments
Assignees
@tyrmullen @mhays-google
Labels
platform:javascript MediaPipe Javascript issues

Comments

@ionif
Copy link

ionif commented Dec 23, 2020

Hello all,

I have a project using Mediapipe Hands on iOS and I've been trying to update from the tfjs model to the new Mediapipe api but even when I enable WebGL2, it still fails to work. I've made sure I'm asking permission using navigator.getmedia properly. Wondering if anyone has any ideas on what's going wrong.

Here's the codepen that I'm testing: https://codepen.io/aionkov/pen/MWjEqWa

Here's the console:

[Warning] I1223 11:05:16.032000 1 gl_context_webgl.cc:146] Successfully created a WebGL context with major version 3 and handle 3 (hands_solution_wasm_bin.js, line 9)
[Warning] I1223 11:05:16.034000 1 gl_context.cc:340] GL version: 3.0 (OpenGL ES 3.0 (WebGL 2.0)) (hands_solution_wasm_bin.js, line 9)
[Warning] W1223 11:05:16.034000 1 gl_context.cc:794] Drishti OpenGL error checking is disabled (hands_solution_wasm_bin.js, line 9)
[Warning] E1223 11:05:16.711000 1 calculator_graph.cc:775] INTERNAL: CalculatorGraph::Run() failed in Run: (hands_solution_wasm_bin.js, line 9)
[Warning] Calculator::Open() for node "handlandmarktrackinggpu__handlandmarkgpu__InferenceCalculator" failed: [GL_INVALID_FRAMEBUFFER_OPERATION]: The framebuffer object is not complete.: glCreateShader in third_party/tensorflow/lite/delegates/gpu/gl/gl_shader.cc:50 [type.googleapis.com/mediapipe.StatusList='\n\x84\x02\x08\r\x12\xff\x01\x43\x61lculator::Open() for node "handlandmarktrackinggpu__handlandmarkgpu__InferenceCalculator" failed: [GL_INVALID_FRAMEBUFFER_OPERATION]: The framebuffer object is not complete.: glCreateShader in third_party/tensorflow/lite/delegates/gpu/gl/gl_shader.cc:50'] (hands_solution_wasm_bin.js, line 9)
[Warning] F1223 11:05:16.712000 1 solutions_wasm.embind.cc:585] Check failed: ::util::OkStatus() == (graph_->WaitUntilIdle()) (OK vs. INTERNAL: CalculatorGraph::Run() failed in Run: (hands_solution_wasm_bin.js, line 9)
[Warning] Calculator::Open() for node "handlandmarktrackinggpu__handlandmarkgpu__InferenceCalculator" failed: [GL_INVALID_FRAMEBUFFER_OPERATION]: The framebuffer object is not complete.: glCreateShader in third_party/tensorflow/lite/delegates/gpu/gl/gl_shader.cc:50 [type.googleapis.com/mediapipe.StatusList='\n\x84\x02\x08\r\x12\xff\x01\x43\x61lculator::Open() for node "handlandmarktrackinggpu__handlandmarkgpu__InferenceCalculator" failed: [GL_INVALID_FRAMEBUFFER_OPERATION]: The framebuffer object is not complete.: glCreateShader in third_party/tensorflow/lite/delegates/gpu/gl/gl_shader.cc:50']) (hands_solution_wasm_bin.js, line 9)
[Warning] *** Check failure stack trace: *** (hands_solution_wasm_bin.js, line 9)
[Warning] undefined (hands_solution_wasm_bin.js, line 9)
[Error] Unhandled Promise Rejection: RuntimeError: abort(undefined) at jsStackTrace@https://cdn.jsdelivr.net/npm/@mediapipe/hands@0.1/hands_solution_wasm_bin.js:9:67558
stackTrace@https://cdn.jsdelivr.net/npm/@mediapipe/hands@0.1/hands_solution_wasm_bin.js:9:67737
abort@https://cdn.jsdelivr.net/npm/@mediapipe/hands@0.1/hands_solution_wasm_bin.js:9:41049
_abort@https://cdn.jsdelivr.net/npm/@mediapipe/hands@0.1/hands_solution_wasm_bin.js:9:179948
wasm-stub@[wasm code]

.wasm-function[10471]@[wasm code] .wasm-function[10466]@[wasm code] .wasm-function[10461]@[wasm code] .wasm-function[10458]@[wasm code] .wasm-function[10474]@[wasm code] .wasm-function[515]@[wasm code] .wasm-function[502]@[wasm code]

wasm-stub@[wasm code]
[native code]
SolutionWasm$send
https://cdn.jsdelivr.net/npm/@mediapipe/hands@0.1/hands.js:33:352
Q@https://cdn.jsdelivr.net/npm/@mediapipe/hands@0.1/hands.js:10:295
https://cdn.jsdelivr.net/npm/@mediapipe/hands@0.1/hands.js:11:90
k@https://cdn.jsdelivr.net/npm/@mediapipe/hands@0.1/hands.js:22:322
promiseReactionJob@[native code] (evaluating 'new WebAssembly.RuntimeError(what)')
(anonymous function) (hands_solution_wasm_bin.js:9:41099)
promiseReactionJob
@mgyong mgyong added the platform:javascript MediaPipe Javascript issues label Dec 30, 2020
@rahulscp
Copy link

rahulscp commented Jan 6, 2021

Not just in ios its not working in all mobiles i tested both android and Ios,

even the demo provided is also not working. is there any workaround ?

https://mediapipe.dev/demo/hands

@Adolar13
Copy link

Adolar13 commented Jan 6, 2021 •
edited

Same error in Hand demo not working on galaxy s7 and s10 in chrome browser. Firefox is working.

@rahulscp
Copy link

any update on this issue ?? still not working in all major mobiles

@anderium
Copy link

The Javascript API also does not work for Safari 14.0.2 (14610.3.7.1.10) on Mac OS X Mojave (10.14.6)

However, this might be unrelated as it only produces a single error in a different function than above.
This is the error I'm getting:
The error in Safari on Mac OS X

It does work in Chrome on the same device.

@jjeon5
Copy link

jjeon5 commented Jan 15, 2021

The Javascript API also does not work for Safari 14.0.2 (14610.3.7.1.10) on Mac OS X Mojave (10.14.6)

However, this might be unrelated as it only produces a single error in a different function than above.
This is the error I'm getting:
The error in Safari on Mac OS X

It does work in Chrome on the same device.

I'm getting the same error on Safari.

@wehriam
Copy link

wehriam commented Jan 19, 2021

The texStorage2d method is not supported in Safari.

It appears the method is referenced in WASM builds such as @mediapipe/hands@0.1/hands_solution_wasm_bin.js.

There are a few references to the method:

@mcclanahoochie - Thank for you and your team's work on this project, and for responding to issues like this. Any updates on Mobile Safari support?

@mhays-google
Copy link
Collaborator

mhays-google commented Jan 19, 2021 via email

@stevestavropoulos
Copy link

Doesn't work on Chrome/Linux too. Firefox is the only working browser right now.

@mhays-google
Copy link
Collaborator

mhays-google commented Feb 4, 2021 via email

@stevestavropoulos
Copy link

Also doesn't work on Chrome/Win10 and Edge/Win10. Firefox is working on both Windows and Linux.

@wumpa-wumpa
Copy link

For facemesh, on Android I get triple the framerate with mediapipe over tensorflow. Hopefully mediapipe on iOS would show similar performance gains and make mediapipe a robust solution for mobile webAR experiences. Any workarounds or idea of a timeline for iOS Safari?

@tyrmullen
Copy link
Collaborator

I think there are at least two separate issues here:
(1) A lot of the incompatibility issues with Safari are related to WebGL2 support. Safari traditionally has only supported WebGL1, and their WebGL2 support is hidden behind a developer toggle. Furthermore, their WebGL2 support is incomplete, failing a large percentage of the official "conformance test" suite (https://www.khronos.org/registry/webgl/sdk/tests/webgl-conformance-tests.html).

As a workaround, many of our older web demos contain patches which will let them run on Safari when the experimental WebGL2 toggle is enabled, but some of the newer CodePens do not have similar patches yet. We were hoping to instead patch them so they run with WebGL1 alone and don't even need an experimental toggle. But for future versions of Safari, all of this may be unnecessary, as the latest Safari Technology Preview (https://developer.apple.com/safari/technology-preview/) appears to be aiming for full WebGL2 support built-in.

In any case, this is an area of active investigation, and Safari support should definitely be on the near horizon.

(2) Our initial ML inference was built for Desktop Chrome; on mobile devices (both iOS and Android), some of the stricter device limits cannot handle the original code paths and limits chosen. I see no reason why they shouldn't run on Linux, OSX and Windows Chrome, so if they're failing to run there, that seems like something to look into, and worth filing a separate issue for tracking.

For mobile, there are several possibilities we're looking into, so please bear with us on that front. I believe this is a similar issue to both
#1430 and #1436.

@delebash delebash mentioned this issue Feb 18, 2021
Open
@narolski
Copy link

narolski commented Mar 4, 2021 •
edited

@mhays-google Is there any update regarding this issue? Your demos are quite amazing, but the lack of the iOS/Safari support limits potential applications significantly 😞

@sgowroji sgowroji mentioned this issue Mar 9, 2021
Closed
@akshatbjain
Copy link

Hey guys! Any update on this issue? Really great work on building mediapipe. Unfortunately, it not working on iOS is a bit of a bummer. Would really appreciate it if you guys could take a look at solving this issue.

@futuyao futuyao mentioned this issue Mar 30, 2021
Closed
@vKrypto
Copy link

vKrypto commented Mar 31, 2021 •
edited

Mediapipe JS solution is being updated day by day, they are working on it..

till then you can use this code (3 march version )

https://www.ornaz.com/realtime/v0_3_march/

or find this code >>

var ASM_CONSTS = {393919: function() { in file hand_solution_wasm_bin.js and replace it with var ASM_CONSTS = {401815: function() {

or

@puny-d
Copy link

puny-d commented Apr 7, 2021

Hi, any update on this issue ?

@nejurgis
Copy link

nejurgis commented Apr 7, 2021

Mediapipe JS solution is being updated day by day, they are working on it..

till then you can use this code (3 march version )

https://www.ornaz.com/realtime/v0_3_march/

or find this code >>

var ASM_CONSTS = {393919: function() { in file hand_solution_wasm_bin.js and replace it with var ASM_CONSTS = {401815: function() {

or

Unfortunately neither of the options you proposed works

@mhays-google
Copy link
Collaborator

mhays-google commented Apr 7, 2021 via email

@tyrmullen tyrmullen mentioned this issue Apr 19, 2021
Closed
@merrijo merrijo mentioned this issue May 27, 2021
Closed
@tyrmullen
Copy link
Collaborator

Most of the groundwork has now been finished to allow JS Solutions to run on Desktop Safari-- you can preview these efforts in the "Holistic" demo, which I believe should now be runnable on Desktop Safari via WebGL 1. The other demos should work too, provided you're running on Solutions API version 0.3 or beyond (so in the "Hands" CodePen, replace any "hands@0.1" instance with "hands@0.3"; I believe there is one instance in the JS and several more in "Settings"-->"HTML"-->"Stuff for ").

Even for most of the failing CodePen demos, enabling experimental WebGL2 usually makes them work for me (running Safari 14.1 on an old MacBook Plus).

TL;DR: I believe Desktop Safari (and hence WebGL1 compatibility) is already (unofficially) supported. However, mobile support has not been looked into just yet, so there will likely still be some issues to work out in order to bring everything to iOS.

@jadams777
Copy link

iOS Chrome and Safari support is critical for my MediaPipe Pose application. Would really appreciate your help adding support for these browsers. Thank you.

@tyrmullen
Copy link
Collaborator

One possible solution (at the potential cost of some performance) would be to run the ML CPU-side instead of GPU-side, since it's only GPU-side ML which encounters issues on iOS at the moment. We hope to have a nice option for this sort of toggling in a few weeks.

@jadams777
Copy link

Hi, does this release include support for iOS? I don't see a changelog. Thank you.
https://www.npmjs.com/package/@mediapipe/pose/v/0.4.1624666670

@tyrmullen
Copy link
Collaborator

No-- Desktop Safari is supported, but the toggling API I mentioned has not yet been finished, so iOS will still have issues. I will try to get a workaround patch into the next release. Will ping threads if/when I submit one.

@sanamumtaz
Copy link

I tested on iPhone Safari. Latest updated version.

The demo stays in loading state.
IMG-20210816-WA0014

@ionif
Copy link
Author

ionif commented Aug 16, 2021 via email

@mhays-google
Copy link
Collaborator

mhays-google commented Aug 17, 2021 via email

@AmitMY
Copy link

AmitMY commented Aug 17, 2021 •
edited

iPhone 11 Pro here.
The codepen demo captures a single shot from my camera, and repeatedly runs pose estimation on it (not moving).
This is consistent with both the front and back cameras.
Performance - when I am in frame, 4-5 fps. out of frame, 9 fps.
(tested in Safari and Chrome, iOS 15)

@mhays-google
Copy link
Collaborator

mhays-google commented Aug 17, 2021 via email

@jadams777
Copy link

Hi Michael, are you able to get the pose tracker running on iOS? I visited this codepen on my iPhone 12, but it didn't track my body. The controls cover the screen, so it's hard to see what's happening. Thanks.

https://codepen.io/mediapipe/pen/jOMbvxw

@AmitMY AmitMY mentioned this issue Aug 18, 2021
Open
@tyrmullen
Copy link
Collaborator

At present, we're prioritizing functionality of the base API over performance and demo fixes, so no ETA for any of the other issues, but we'll be looking into fixing the selfie-segmentation API shortly.

Just to summarize, on most iPhones the current state of the demos should be the following:

  • Demos except selfie-segmentation should work*
  • Webcam should work with latest control_utils update**
  • Pre-canned videos (like for Objectron) should work
  • Uploading a video file might still fail for some devices
  • Technically there may be a slight loss in precision for mediapipe::Tensors when used on iOS, but in practice I've seen no negative impact as of yet***
  • We now automatically switch over to using CPU inference on iOS devices
  • The rendering is still being performed with WebGL1, as Safari still does not yet support WebGL2 (hopefully soon this may change?)
  • Performance is much slower, at least in part as a result of using CPU inference, but this has not been extensively investigated/profiled yet
  • There are quite a few calculators unique to selfie-segmentation which we need to ensure iOS-compatibility for; also it's the only API which outputs two textures.

** The "stays loading and never works" issue is usually this-- you can tell if this is the problem you're running into because the console log will give you a "Permissions denied" error, even though you accepted the camera permissions popup.

*** One workaround I applied lets them sometimes use 16-bit floating point textures rather than 32-bit ones, as iOS devices often don't support the WEBGL_color_buffer_float extension.

@jadams777 Pose seems to be working for me (at a few fps) on an iPhone 6S with Safari. I use the "full" demo so I can see more of the display: https://codepen.io/mediapipe/full/jOMbvxw, but I agree, it would be nice for iOS if we could scale back the controls (or maybe just add a toggle to show/hide those potentially?).

@AmitMY I have not seen this issue myself, but I've run into issues with similar symptoms in the past with iOS-Safari, where the pipeline is running, but the current video frame texture being passed back from the webcam stream isn't being updated by the browser. IIRC, this could happen when the browser thinks the video isn't being used, and so tries to throttle the rendering for it. As a test, perhaps try see if displaying the video directly on the webpage fixes the playback issue for you (although of course that will add an unnecessary extra rendering cost). And in any case, let's file this as a separate bug if it persists for you, since I believe it's unrelated to the other issues here.

@tyrmullen tyrmullen self-assigned this Aug 18, 2021
@jadams777
Copy link

jadams777 commented Aug 18, 2021 via email •
edited

@jadams777
Copy link

I tried the Pose codepen on my iPad Pro 2021, and there is a weird flashing effect. Check it out:
https://photos.app.goo.gl/hAx5wRMavc2gTa4H8

@AmitMY
Copy link

AmitMY commented Aug 19, 2021 •
edited

The rendering is still being performed with WebGL1, as Safari still does not yet support WebGL2 (hopefully soon this may change?)

WebGL2 is available on iOS 15. Betas are currently available, public release in 1 month. https://caniuse.com/webgl2

@peter-wildman
Copy link

I've recently had this error emerge in Safari on Desktop. Is there a known solution:
Screen Shot 2021-08-20 at 1 51 23 pm

@jadams777
Copy link

jadams777 commented Aug 20, 2021 via email

@mhays-google
Copy link
Collaborator

mhays-google commented Aug 20, 2021 via email

@jadams777
Copy link

jadams777 commented Aug 20, 2021 via email •
edited

@tomhsiao1260
Copy link

tomhsiao1260 commented Aug 23, 2021 •
edited

Hi, everyone.

The current CodePen pose demo works well on my iPhone12 device.
https://codepen.io/mediapipe/pen/jOMbvxw

However, I found that this demo use controls.SourcePicker in control_utils npm package to open the video. In some use cases, I think it's more convenient to use camera_utils instead to open the video like the following document does.
https://google.github.io/mediapipe/solutions/pose#javascript-solution-api

However, after testing, it seems that camera_utils still cannot support IOS device (the latest npm update was 4 months ago). Just want to ask, are there any plans about the camera_utils IOS support for developers who don't need a control panel in their application?

@mhays-google
Copy link
Collaborator

mhays-google commented Aug 23, 2021 via email

@tomhsiao1260
Copy link

tomhsiao1260 commented Aug 24, 2021 •
edited

Thanks for clarifying. Also, the solution works. Really appreciate!

@tyrmullen
Copy link
Collaborator

Looked into the selfie-segmentation issue a bit, and found what I believe to be some texture memory handling issues. I don't have the bandwidth to fully fix the underlying problems at the moment, but I created a patch to at least allow selfie-segmentation to run on iOS (tested successfully on my old iPhone6S). We'll try to get this integrated into the JS Solutions codebase as soon as possible.

@sanamumtaz
Copy link

Looked into the selfie-segmentation issue a bit, and found what I believe to be some texture memory handling issues. I don't have the bandwidth to fully fix the underlying problems at the moment, but I created a patch to at least allow selfie-segmentation to run on iOS (tested successfully on my old iPhone6S). We'll try to get this integrated into the JS Solutions codebase as soon as possible.

Kindly update about the current status.

@tyrmullen
Copy link
Collaborator

The team is still discussing how best to conditionally apply the patch. Will update again when I have something more substantive.

@tyrmullen
Copy link
Collaborator

The team desired a specific form for the patch; I have now refactored everything accordingly and submitted the patch. The next update to the selfie-segmentation JS Solution will contain the fix. All other iOS Solutions will also have slightly altered behavior with their next updates, but the impact there will hopefully be minimal/negligible.

Also, it should be noted this issue was NOT iOS-specific, but rather occurred due to using the selfie-segmentation module with CPU ML inference. So in case it helps other people running into similar odd selfie-segmentation issues, the workaround patches were all just different ways to force the TensorsToSegmentationCalculator to use the "use_gpu=true" path despite ML being performed on the CPU (and thus having mediapipe::Tensor objects with only CPU backing by default).

@tyrmullen
Copy link
Collaborator

On an unrelated note, @jadams777: do you mind trimming the console output from your last message a bit (or replacing it with a link)? The length of the resulting post makes it difficult to scroll through the previous responses for full thread context.

@tyrmullen
Copy link
Collaborator

While our GPU ML does not run on iOS, so performance will be slower, I think with the next update, all MediaPipe CodePens should at least run on iOS Safari. So closing out this bug as fixed.

@sgowroji sgowroji removed the stat:awaiting googler Waiting for Google Engineer's Response label Sep 14, 2021
@cbasavaraj
Copy link

cbasavaraj commented Nov 10, 2021 •
edited

Hello, I have a question on Camera vs SourcePicker. At the moment, SourcePicker isn't working on iPhone 12, but Camera is, after adding playsinline="true" crossorigin="anonymous". But Camera doesn't allow changing the size of CanvasElement on the fly, for example, if the phone is turned by 90 degrees. Anyway to get the size field while using Camera?

Source Picker:

new controls.SourcePicker({
      onFrame:
        async (input, size) => {
          resolution = getResolution(size);
          canvasElement.width = resolution.w;
          canvasElement.height = resolution.h;
          await faceMesh.send({image: input});
        },
    }),

Camera:

const camera = new Camera(videoElement, {
  onFrame: async () => {
    await faceMesh.send({image: videoElement});
  },
  width: resolution.w,
  height: resolution.h
});

getResolution:

function getResolution(size) {
  let aspect = size.width / size.height;
  let width = null, height = null;
  if (window.innerHeight < window.innerWidth) {
    height = window.innerHeight;
    width = height * aspect;
  }
  else {
   width = window.innerWidth;
   height = width / aspect;
  }
  return {
    w: width,
    h: height
  }
}

@AHMetaCubes
Copy link

Status on working on mobile browsers?

@hktalent
Copy link

https://51pwn.com/3dDetect.html
me too
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tyrmullen tyrmullen

@mhays-google mhays-google

Labels
platform:javascript MediaPipe Javascript issues
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests