Javascript Solutions Source Code

[Choons]

I'm happy to see that a javascript solution has been released for the facemesh, but can't seem to find the uncompiled javascript or typescript source code here on GitHub. Am I just missing it, or is it not posted?

[mgyong]

We don't plan to release the source code for MP JS Solutions API. Python source is out but not JS for now

[Choons]

?? if I'm not mistaken the python code is available? Anyway, without the code, then we need more documentation on the API than the little example reveals. I can't figure out the range of capabilities from that alone.

[djthegr8]

We don't plan to release the source code for MP JS Solutions API. Python source is out but not JS for now

@mgyong why would that be though? The files can certainly be improved by open sourcing and anyways, people can access the minified code, so no scope for proprietary right?

[djthegr8]

And regarding documentation, please review my PR #1434 which adds helpful notes about API and Utilities

[afogel]

any update on this? @mgyong (might have gotten lost b/c most people were probably out of the office around christmas/new years)

[afogel]

@mgyong @tyrmullen sorry to ping, just curious whether there's any new information about getting better documentation/more information about why MP doesn't plan on releasing the source for the JS Solutions API?

[BSchrift]

Even if the source code has not been released, has anyone documented the API or created typescript declarations?

[mgyong]

The problem is that we require bazel to support the --wasm option and additional work on our end to clear up our internal code in order to release. Currently there are no plans till maybe Q2 2021.

[Choons]

it's a bit silly really considering the code is already out there in the tensorflow.js models examples. The question is how similar or different your release is from that?

…

On Mon, Mar 29, 2021, 6:34 PM Ming Yong ***@***.***> wrote: The problem is that we require bazel to support the --wasm option and additional work on our end to clear up our internal code in order to release. Currently there are no plans till maybe Q2 2021. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1408 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD5GFYOEHBUX6WIVIOS2MMDTGEE77ANCNFSM4VB2C4XQ> .

[mgyong]

@Choons It's quite different. tf.js does not use wasm. MP js solutions api does. If it was easy for us to release, we would have done it. Matter of bandwidth and priority

[Choons]

true. it doesn't expose the killer features promised in the write-up. quite different.

…

On Mon, Mar 29, 2021 at 6:57 PM Ming Yong ***@***.***> wrote: @Choons <https://github.com/Choons> It's quite different. tf.js does not use wasm. MP js solutions api does. If it was easy for us to release, we would have done it. Matter of bandwidth and priority — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1408 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD5GFYP5WWPHTJD46UITKNDTGEHUZANCNFSM4VB2C4XQ> .

[chuoling]

One thing we could do quickly is to include the TypeScript declarations (w/ inline comments) in the NPM package. @Choons would that help?

[Choons]

Yes! Anything you can pass over to us would be great. We're willing to help!

…

On Mon, Mar 29, 2021 at 7:36 PM chuoling ***@***.***> wrote: One thing we could do quickly is to include the TypeScript declarations (w/ inline comments) in the NPM package. @Choons <https://github.com/Choons> would that help? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1408 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD5GFYLQADEG32JVKNGWKRDTGEMKFANCNFSM4VB2C4XQ> .

[chuoling]

Great. @mhays-google owns our JS solutions and will look into it.

[Choons]

I'll leave this here. https://github.com/spite/FaceMeshFaceGeometry

maybe people will find it useful

[mhays-google]

I am uploading our TypeScript exports file as part of the npm package for each and every solution. For any solution, the easiest way (I've found) to get to this file will be:

https://cdn.jsdelivr.net/npm/@mediapipe/camera_utils/index.d.ts
https://cdn.jsdelivr.net/npm/@mediapipe/control_utils/index.d.ts
https://cdn.jsdelivr.net/npm/@mediapipe/drawing_utils/index.d.ts
https://cdn.jsdelivr.net/npm/@mediapipe/face_mesh/index.d.ts
https://cdn.jsdelivr.net/npm/@mediapipe/face_detection/index.d.ts
https://cdn.jsdelivr.net/npm/@mediapipe/pose/index.d.ts
https://cdn.jsdelivr.net/npm/@mediapipe/hands/index.d.ts
https://cdn.jsdelivr.net/npm/@mediapipe/holistic/index.d.ts

...et. al.

Even if you are not familiar with TypeScript, this should be pretty decipherable.

While this is not full documentation, per se, it should give you visibility into what can be called within the API. This is a quick fix to address what is obviously a very opaque interface. I will add a README.md to the packages giving these same directions shortly, but I wanted to have this to you by end of day. I'll look into better documentation in general.

[mhays-google]

Regarding the full release of the source code, this is being discussed, I personally support it, but there are a few hurdles.

Hopefully releasing the index file solves a few immediate issues -- thank you for the patience with that, by the way. And we'll look into what it will take to release more of the code (or at least the unobfuscated javascript).

[Choons]

Much appreciated, Michael! Yeah I love web assembly as a concept, but without documentation of what the wasm modules expose, it's just shooting in the dark trying to guess what's in the API. Also love the concept of using emscripten/llvm on c++ to create javascript/wasm, but in practice I have found the reality is often more difficult to create than just porting the code manually to javascript or Typescript, or Assemblyscript and then to wasm for the performance gains. Maybe the community here can contribute in those areas of the API that are lacking due to the work load of the developers.

[afogel]

THIS IS AWESOME!! Thanks @mhays-google

[BSchrift]

Thanks @mhays-google! This is so helpful.

I do have one more quick question: is there a reason why the package.json for the npm package doesnt specify a "main" or "typings" field?

It would be great to be able to say, for example, import { Hands } from '@mediapipe/hands' and have typescript know where to get the js code and the associated type declarations. Looking at the distributed package.json I think it would just be a matter of setting:

"main": "hands.js", 
"typings": "index.d.ts",

or the equivalent in each package, unless I'm misunderstanding how the packages are structured.

[lostfictions]

It's really unfortunate that there's no better documentation available, especially for the helper libraries (@mediapipe/drawing_utils, @mediapipe/camera_utils, @mediapipe/control_utils). Without them, it's remarkably hard to figure out how one would go about implementing drawing code -- for example, drawConnectors is totally opaque since it's shipped as a Closure-minified module, and there's no explanation or documentation elsewhere for how landmark indices are meant to be interpreted. As it stands, MediaPipe JS seems to either be a toy to show that a browser implementation is theoretically possible, or... a solution non-googlers are supposed to reverse-engineer?

[liufsd]

+1

[ayoub-root]

any updates ?

[alexandernst]

Almost hitting the 2 years mark and still no source code. Google is totally misleading everybody when stating that this is open source. Either release the code or clearly state that it's not open source.

[wheelie33]

lease the source code for MP JS Solutions API. Python source is out but not JS for n

Why?

[wheelie33]

ap as input if you really needed to (this was discussed in anothe

Can you provide a working example of the HTMLImageElement? I'm trying a VERY VERY small prototype and it has some unhandled exceptions.

[esinanturan]

Regarding the full release of the source code, this is being discussed, I personally support it, but there are a few hurdles.

Hopefully releasing the index file solves a few immediate issues -- thank you for the patience with that, by the way. And we'll look into what it will take to release more of the code (or at least the unobfuscated javascript).

@mhays-google Any new regarding the release ?

[kuaashish]

@Choons,

JS Source is available for MP Tasks, And Will not be published for other Solutions.

[afogel]

Tasks

Hi @kuaashish, thanks for the update! Based on my read through the tasks documentation, does that meant that we would effectively create out own holistic model using a tasks pipeline that we define for ourselves?

How much less efficient will it be to use the pose, face, and hands tasks in conjuction, as compared to using the holistic model?
Thanks :)

[Choons]

it's pretty much useless. I moved on to other code a long time ago.

[alexandernst]

So long for the "open source" part of this project

[kuaashish]

@Choons,

Thanks for the confirmation. If this is no longer issue from your end, Can we move ahead and close the this?

[github-actions]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[github-actions]

This issue was closed due to lack of activity after being marked stale for past 7 days.