Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JS: npm package #15315

Open
cancerberoSgx opened this issue Aug 15, 2019 · 7 comments
Open

JS: npm package #15315

cancerberoSgx opened this issue Aug 15, 2019 · 7 comments

Comments

@cancerberoSgx
Copy link
Contributor

@cancerberoSgx cancerberoSgx commented Aug 15, 2019

I see there's no package for opencv.js so JavaScript users can install it using npm install as is accustomed by JS users.

The benefits of this would be also that is better integrated with web bundlers like webpack, parcel, browserify which simplify development workflow used in generally for web development.

I think for supporting this, the .wasm version could be better. BTW the current open.js file, although it works since it supports UMD, it would not be the best since it will be combined by bundlers (so the resulting .js file will contain it and will be huge bloking the user's apps). Instead it should be downloaded separately optionally allowing users to define an alternative url for it (like a CDN).

I'm working on a proof f concept for this, and also to support / tests for node.js as API and a little CLI tool too, and I wonder if something like this would be accepted as a PR to at least start the discussion.

Thanks

@frastlin

This comment has been minimized.

Copy link
Contributor

@frastlin frastlin commented Aug 20, 2019

Is there a way to split the open.js file into modules so one can take advantage of tree-shaking?

@cancerberoSgx

This comment has been minimized.

Copy link
Contributor Author

@cancerberoSgx cancerberoSgx commented Aug 21, 2019

I have feedback I would like to work on.

Initial slowdown because base 64 decoding of a string, not because of size.

  • The bottle neck is the initial load and I don't think it's because the file size but because the time taken on decode the big base64 embedded string. I think this is really bad design
  • I'm comparing against my ImageMagick own port that weights 9Mb and include many libraries that are all statically linked, some: fftw freetype jng jp2 jpeg lcms png raw tiff webp zlib .demos a project:
  • https://cancerberosgx.github.io/demos/magica/playground/
  • https://cancerberosgx.github.io/demos/magica/canvas/
  • (project page)https://github.com/cancerberoSgx/magica
  • Even if the file weights 20mb there are ways to improve that situation but the current base64 initial decoding is un-remediable (and imo unneccesary - I think it's only there because the js tutorials loads it like that). I wonder why is it compiled like that ? is it really needed?

WebAssembly output is polluted and there's no way to get the "clean" .wasm /asm.js output files

  • I don't like the magick behind "generate-umd" thing to work needs -s SINGLE_FILE=1 which prevents to build .wasm version

  • The build should be clean and simple generating opencv.wasm and opencv.js (or only opencv.js if user chooses WASM=0).

    • Then, if authors, prefer to use UMD or add custom browser utilities then do it in a copy of those files but please leave a clean build for users with other needs / technologies.

helpers.js and cv.imread

  • helpers.js pollutes the wasm build with DOM APIs . Fortunately these don't interfere (at runtime) with non DOM platforms like node.js but:
  • it overrides the well-known public cv.imread and totally changes the signature and semantics. Why? It prevents using it in node.js. The same advice as before, don't modify/override please, just create new files or signatures. In the browser this function still could be useful and I would say needed when using emscripten FS API and another image format encoder/decoder than the canvas.(user's could use mentioned ´magica`project to load images, or ´jimp´- there are plenty image decoders supporting browser and node.
  • non canvas users are forced to use for example cv.matFromImageData() which is suboptimal wince it requires memory copy while imread() is designed to stream from filesystem. Remember in node.js the fileystem is the real one and that method was designed just for that!.
  • This is an example of how I'm loading images using jimp (node.js) (https://github.com/cancerberoSgx/mirada/blob/master/mirada/test/loadTest.ts) ?
  • This manual extra programming instead of using FS could be painful when referencing several images...
  • I?m working on a proposal (kind of a proxy API) to de-acoplate the image decoding from the library by just implementing an interface to adapt a library/technology.

wasm build

  • It's very hard or impossible to pass extra emscripten arguments, like -03 or tracing / debugger. It would be ideal if users could pass it as arguments to build_js.py

  • There are some very interesting regarding math, concurrency, memory. Also it would allow contributions / configurations so the build could include libraries like libpng, etc (if configured). so the library has support for minimal formats and gets DOM agnostic.

  • my attempt to build the ideal opencv.js npm package: I'm working on what for me would be an ideal opencv.js npm package here https://github.com/cancerberoSgx/mirada/. The idea is to use the official opencv.js wasm build files, support node.js and the browser with the same API which should be small, ideally only a thin layer with almost no implementation at all that directly represent vc.* methods. Probably some separate modules to easy image format handling and JS / asynchronous syntax sugar. The hard work and most important would be to provide TypeScript typings matching this API hopefully directly / mechanically ported from doxygen apidocs (working on that) . Also some tests . But keep the implementation minimal and agnostic on any library/technology

Of course the final idea is to contribute all of this to this project but right now I prefer to play and see which concrete parts I can push or are blockers for this, Help is welcome

@cancerberoSgx

This comment has been minimized.

Copy link
Contributor Author

@cancerberoSgx cancerberoSgx commented Aug 29, 2019

Update: I implemented another Playground project but this time allowing users to experience TypeScript editor with autocompletion, type-checking and online docs (consuming the TS output of mirada) :

https://cancerberosgx.github.io/demos/mirada-ts-playground (tested only in chrome and firefox so far)

It uses monaco-editor (the same as vscode's) and consumes typescript declaration files that are automatically generated from opencv sources by this auxiliary tool of mine:

https://github.com/cancerberoSgx/mirada/tree/master/doxygen2typescript

It currently consumes doxygen xml output. The only hack I made was to filter the content using bindings.cpp since the docs are huge and the js biuld only support a fraction of it. The rest mostly is XML paring, templates and a a couple of TypeScript AST manipulations at the end.

These are examples of the generated typescript declarations processed with jsdoc rendering tools - I didn't put much effort on them since I'm only interested in the editor/type checking experience but just for you to see the current status/quality:

Although there are some details still, (mostly with references, enum types and formula rendering) I'm quite happy with it and now I'm it'm using it successfully to learn the library's API since I'm actually totally new with it) . Probably in editor-consumable documentation/type-declarations (which is my target) I should use shorter descriptions, but for now (and for me) I'm dumping the whole thing.)

I'm porting examples while I learn and use them library tests/ CI or in the playground and I must say the port of examples in general is quite straight forward (and surprisingly easier from C++) - I think I cover in most of the use cases shown in the tutorials. Nevertheless I will keep adding more as I learn.

As I said, I have suggestions for the js build that I will try to translate to PRs, right now I prefer to learn and enjoy a little bit more before that, and wanted to give this update just in case anybody has some opinions / feedback. I'm sorry for taking this issue mostly to discuss with myself before I have a PR proposal or a concrete contribution. Thanks

@frastlin

This comment has been minimized.

Copy link
Contributor

@frastlin frastlin commented Aug 29, 2019

This is really awesome, do you think you could do an example with SolvePnP and maybe Aruco markers?

It would also be really nice if we could use this in both the browser and React Native or Native Script.

@cancerberoSgx

This comment has been minimized.

Copy link
Contributor Author

@cancerberoSgx cancerberoSgx commented Aug 30, 2019

This is really awesome, do you think you could do an example with SolvePnP and maybe Aruco markers?

Sure if build supports them I will try. I recently attempted to add the module ml (machine learning) in the js build and although it is included by cmake it is not published in the global Module variable. I wonder if you know about any documentation about this topic since I'm kind lost here and would be awesome to try ml and other more training-oriented modules in the browser and node.js.

It would also be really nice if we could use this in both the browser and React Native or Native Script.

I'm running the tests in node.js - about those two technologies I don't really know them, but if they are oriented toward small devices (and performance/battery is important) I would use the c++/python versions not this one. Specially if you execute it as a command line since

Also since this project has so little JS implementation it would be interesting to try https://docs.assemblyscript.org/ or similar to generate also compile the browse clue code r and user's as webassembly altogheter. I will try to test things like https://github.com/as2d/as2d to see how/if this kind of things can be (ever used them) although I don't know if these things are related to React Native or Native Script - will let you know, thanks

@cancerberoSgx

This comment has been minimized.

Copy link
Contributor Author

@cancerberoSgx cancerberoSgx commented Sep 1, 2019

Is there a way to split the open.js file into modules so one can take advantage of tree-shaking?

Although I never tried, emcc has some support for dynamic linking, They say is still experimental and has some impact on the speed : * https://emscripten.org/docs/compiling/Building-Projects.html#dynamic-linking * https://github.com/emscripten-core/emscripten/wiki/Linking

  • I have some experience tuning performanceI in the browser and int this case personally, I would not sacrifice speed just to reduce the output size. 1 MB more or less won't be noticed and there are other more efficient ways to alleviate this problem..

Then, emcc also supports this, which again I have no experience with: https://emscripten.org/docs/optimizing/Optimizing-Code.html#very-large-codebases ? what's your concern ? code size ? Do you think tree shaking you mention could improve also speed in this case? I have previous experience but only with WASM before, and here I suspect parsing such a big JavaScript is expensive for the browser. I tried without luck to build the .wasm I think should improve parsing at least, but I didn't had much time. Also I notice emcc uses only -O2 and -NO_exceptions by default. I wonder why not -O3 and if somebody tried the other optimization flags. Do you now if there is some benchmark that makes sense in JS to compare ?

Also a question, ImageMagick which targets a similar area, has two important compile flags that let you trade off between processing speed, memory and image output quality / match precision.
One is the quantum which is the depth for color channels and this affects memory consumption- n the past it used 8 bits but now several image formats support 16 and 32 . (now their default is 16) . The other flag is called HDRI that is related with Math and gives much better results in those filters / transformations involving complex arithmetic like like space domain transformations/ FFT, etc I'm not an expert but some operations needs that enabled or they won't work

The first flag impacts memory while this second flag impacts speed. just in case I want to know if there are any similar settings in opencv build ?

Thanks

@frastlin

This comment has been minimized.

Copy link
Contributor

@frastlin frastlin commented Sep 2, 2019

I wonder if you know about any documentation about this topic since I'm kind lost here and would be awesome to try ml and other more training-oriented modules in the browser and node.js.

There is an issue working through adding new modules to the build. That is about SolvePnP and some other calib modules, but if one is able to add those, then the ML modules shouldn't be that much different.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
3 participants
You can’t perform that action at this time.