info: MediaPipe javascript api

[delebash]

MediaPipe now has it's own javascript api that you can use instead of tensor flow.

https://ai.googleblog.com/2020/12/mediapipe-holistic-simultaneous-face.html

https://google.github.io/mediapipe/getting_started/javascript#ready-to-use-javascript-solutions

[vladmandic]

Thanks - I'm impressed how they implemented smoothing.
And performance is definitely higher than with human.

Unfortunately, cannot use it to learn anything useful as @mediapipe solutions are written in C++ (JS is used just as loader for a custom compiled WASM binary).

I do see that @mediapipe uses tflite models which are normally not even supported on desktop - this is all custom work by @mediapipe and does not rely on a published release of tensorflow.
Even image handling is done using OpenCV binary embedded in @mediapipe WASM module.

if anything, @mediapipe showcases just how much faster is a native binary code than any JS code that is interpreted in browser and uses a generic framework for backend (which is not unexpected).

[vladmandic]

i'm converting this to a discussion topic now that github supports discussions :)

[ButzYung]

I remembered I came across some discussions about the doubts in the future of TFJS, especially with the rise of WASM. I wonder if this really means a JavaScript solution of Tensorflow is not the way to go, or are there still some benefits of using JS for Tensorflow despite the performance disadvantage?

[vladmandic]

End-to-end WASM solution is faster than JS, there is no question about it.

But what you mention (I saw that discussion) is a bit misleading as TFJS already fully supports multiple backends including WASM and WebGL.

So how can the question be TFJS vs WASM when TFJS can use WASM?

So why is @mediapipe so much faster than TFJS+WASM?
Because it's end-to-end written in C++ and compiled to WASM.
TFJS is a framework, so there is a lot of interaction between JS code and WASM (if chosen as backend) and that interaction can be more costly than actual processing inside WASM. Plus the implementation in WASM has to be as generic as possible (again, it's a framework) and cannot optimize for a single case (such as @mediapipe).

On the other hand WASM is superior solution for quick models - it has very low latency. But GPU integration within WASM is as difficult as it can be - I haven't seen much of it so far. So for complex models, WebGL is still a superior backend.
Think of it this way - if a model is doing real-time video analysis, WASM is great. But if model is doing some serious work measured in minutes instead of milliseconds, forget about WASM.

Now, WebGL does have an issue of double translation (most browsers implement WebGL via DX first and then using actual GL calls later) plus much higher GPU memory usage.

I am looking with anticipation of WebGPU standard that would allow JS to do GPU processing more directly. It's currently in experimental stage (can only be enabled on dev version of chrome), but TFJS team is actively working on it already.

[vladmandic]

To add, I'm working on developing some models of my own (not related to human) - actual model training with WASM is a nightmare, it's not even an option right now.

Btw, if you want absolutely maximum performance for ML, use C++ and link to CUDA libraries for GPU acceleration.
No JavaScript (browser or NodeJS), no WASM, no WebGL, no WebGPU, etc.

And then use a higher level language to interact with library you wrote in C++/CUDA. But that language itself should support parallel execution - and JS doesn't (it's single-threaded by design).

Python would be most popular because and honestly, it's not bad (horrible native performance, but can use external libraries with minimum overhead).
Ideally, I'd look at something like Julia for optimal high-level performance.

[ButzYung]

Yeah I know it's not fair to compare TFJS with WASM directly, since just like what you said, TFJS has a WASM backend as well. But assuming that you need to have the best performance on a web platform, this kind of all-WASM solution (obviously GPU accelerated as well in this case) saving all the slower JS preprocessing may be the way to go.

PS: Been working on my own project that use TFJS + Facemesh + PoseNet + Handpose. Just when I got it working, then I came across your human project by luck which obviously had some advantages over my solutions. And then this MediaPipe API... really need to look into this LOL. At the very least I need the multiple hand detection since the TFJS Handpose can't do it for now. Also wonder if there are any accuracy differences between the TFJS and the MediaPipe solutions.

[vladmandic]

But assuming that you need to have the best performance on a web platform, this kind of all-WASM solution saving all the slower JS preprocessing may be the way to go.

Absolutely. And it's not just about ML - I know several graphic-heavy websites that that converted from JS to WASM and got massive performance boost. But that is not for casual developer for sure.

Current state of WASM compilers (e.g. emscripten) is getting better, but it will be a while since they can be used as "just works". I'm looking forward to time when normal JS or Python code can be compiled to WASM, but that's not even on a horizon as WASM cannot deal with dynamical types - everything has to be very strictly typed (even generics in C++ are a no-go).

At the very least I need the multiple hand detection since the TFJS Handpose can't do it for now

Yeah, took me a while to get that one working, it's pretty messy. My code diverged from TFJS example quite a lot.

Also wonder if there are any accuracy differences between the TFJS and the MediaPipe solutions

Quite a lot based on a quick initial glance.

• First, models are updated. But @mediapipe only released converted tflite models that cannot be used in TFJS. Given that tflite cannot be used for training, only as end product of conversion, I wish they released original trained models so users can convert them to different types. I'll dig around, maybe I can find something (like I found blazeface back model which wasn't initially released by @mediapipe, only front model.

• Second, @mediapipe does a lot of postprocessing to smooth results between frames. I've tried that in JS and it's just too slow. I may give it another shot, but I don't have my hopes up.

• Third, they do note they have a second way of detecting hand box when first one fails (which is quite often and is the biggest problem with hand model). But no details released so far.

[vladmandic]

one more thought - as a casual developer, I don't even want to think about developing WASM solution other than a proof-of-concept if I still have to think about things like:

• is threading supported in browser or not? It can be enabled, but it's anything but stable.
• is SIMD supported in browser or not? It can be enabled, but it's not by default.
• which sensor classes are available per browser? (e.g., i can't access camera from WASM, I still need to pass data from JS to WASM unless I do all-in browser-specific code as done in OpenCV - I guess that's why @mediapipe uses OpenCV)

WASM promise was that dev will be able to take his existing binary code and convert it run natively in browser - we're still years from that.

[ButzYung]

I tried to use the MediaPipe API in my project, but unfortunately it doesn't seem to support web worker (a must in my case, since there are some intensive 3D animations, and there is little room in the mani UI thread for other CPU-intensive task). So at the end I tried your @vladmandic human library instead, but I encountered some issues.

1. I loaded human.js inside worker (via importScripts), but I have to load the human object as new Human.default() instead of new Human() (I can import human.esm.js as module, but I want to avoid that in web worker).
2. .warmup() doesn't seem to work as worker doesn't have Image.
3. The accuracy of your PoseNet model is low, obviously lower than the default one from TFJS. Is it possible to change some parameters so that it is on par with the TFJS one, or even have the option to load ResNet instead of Mobilenet?

[djthegr8]

Um, just more info, Mediapipe doesn't even implement Multi-Threading. If they do, then that could lead to a very significant FPS increase from the already-awesome fps they have....

[vladmandic]

FYI, I've managed to successfully convert and implement MediaPipe's BlazePose model as alternative to PoseNet.
PoseNet is still the default, but can be switched via configuration options. If interested, check out model notes as it's performance is quite difference.