JS: npm package

[cancerberoSgx]

I see there's no package for opencv.js so JavaScript users can install it using npm install as is accustomed by JS users.

The benefits of this would be also that is better integrated with web bundlers like webpack, parcel, browserify which simplify development workflow used in generally for web development.

I think for supporting this, the .wasm version could be better. BTW the current open.js file, although it works since it supports UMD, it would not be the best since it will be combined by bundlers (so the resulting .js file will contain it and will be huge bloking the user's apps). Instead it should be downloaded separately optionally allowing users to define an alternative url for it (like a CDN).

I'm working on a proof f concept for this, and also to support / tests for node.js as API and a little CLI tool too, and I wonder if something like this would be accepted as a PR to at least start the discussion.

Thanks

[frastlin]

Is there a way to split the open.js file into modules so one can take advantage of tree-shaking?

[cancerberoSgx]

I have feedback I would like to work on.

Initial slowdown because base 64 decoding of a string, not because of size.

• The bottle neck is the initial load and I don't think it's because the file size but because the time taken on decode the big base64 embedded string. I think this is really bad design
• I'm comparing against my ImageMagick own port that weights 9Mb and include many libraries that are all statically linked, some: fftw freetype jng jp2 jpeg lcms png raw tiff webp zlib .demos a project:
• https://cancerberosgx.github.io/demos/magica/playground/
• https://cancerberosgx.github.io/demos/magica/canvas/
• (project page)https://github.com/cancerberoSgx/magica
• Even if the file weights 20mb there are ways to improve that situation but the current base64 initial decoding is un-remediable (and imo unneccesary - I think it's only there because the js tutorials loads it like that). I wonder why is it compiled like that ? is it really needed?

WebAssembly output is polluted and there's no way to get the "clean" .wasm /asm.js output files

• I don't like the magick behind "generate-umd" thing to work needs -s SINGLE_FILE=1 which prevents to build .wasm version

• The build should be clean and simple generating opencv.wasm and opencv.js (or only opencv.js if user chooses WASM=0).

  • Then, if authors, prefer to use UMD or add custom browser utilities then do it in a copy of those files but please leave a clean build for users with other needs / technologies.

helpers.js and cv.imread

• helpers.js pollutes the wasm build with DOM APIs . Fortunately these don't interfere (at runtime) with non DOM platforms like node.js but:
• it overrides the well-known public cv.imread and totally changes the signature and semantics. Why? It prevents using it in node.js. The same advice as before, don't modify/override please, just create new files or signatures. In the browser this function still could be useful and I would say needed when using emscripten FS API and another image format encoder/decoder than the canvas.(user's could use mentioned ´magica`project to load images, or ´jimp´- there are plenty image decoders supporting browser and node.
• non canvas users are forced to use for example cv.matFromImageData() which is suboptimal wince it requires memory copy while imread() is designed to stream from filesystem. Remember in node.js the fileystem is the real one and that method was designed just for that!.
• This is an example of how I'm loading images using jimp (node.js) (https://github.com/cancerberoSgx/mirada/blob/master/mirada/test/loadTest.ts) ?
• This manual extra programming instead of using FS could be painful when referencing several images...
• I?m working on a proposal (kind of a proxy API) to de-acoplate the image decoding from the library by just implementing an interface to adapt a library/technology.

wasm build

• It's very hard or impossible to pass extra emscripten arguments, like -03 or tracing / debugger. It would be ideal if users could pass it as arguments to build_js.py

• There are some very interesting regarding math, concurrency, memory. Also it would allow contributions / configurations so the build could include libraries like libpng, etc (if configured). so the library has support for minimal formats and gets DOM agnostic.

• my attempt to build the ideal opencv.js npm package: I'm working on what for me would be an ideal opencv.js npm package here https://github.com/cancerberoSgx/mirada/. The idea is to use the official opencv.js wasm build files, support node.js and the browser with the same API which should be small, ideally only a thin layer with almost no implementation at all that directly represent vc.* methods. Probably some separate modules to easy image format handling and JS / asynchronous syntax sugar. The hard work and most important would be to provide TypeScript typings matching this API hopefully directly / mechanically ported from doxygen apidocs (working on that) . Also some tests . But keep the implementation minimal and agnostic on any library/technology

Of course the final idea is to contribute all of this to this project but right now I prefer to play and see which concrete parts I can push or are blockers for this, Help is welcome

[cancerberoSgx]

Update: I implemented another Playground project but this time allowing users to experience TypeScript editor with autocompletion, type-checking and online docs (consuming the TS output of mirada) :

https://cancerberosgx.github.io/demos/mirada-ts-playground (tested only in chrome and firefox so far)

It uses monaco-editor (the same as vscode's) and consumes typescript declaration files that are automatically generated from opencv sources by this auxiliary tool of mine:

https://github.com/cancerberoSgx/mirada/tree/master/doxygen2typescript

It currently consumes doxygen xml output. The only hack I made was to filter the content using bindings.cpp since the docs are huge and the js biuld only support a fraction of it. The rest mostly is XML paring, templates and a a couple of TypeScript AST manipulations at the end.

These are examples of the generated typescript declarations processed with jsdoc rendering tools - I didn't put much effort on them since I'm only interested in the editor/type checking experience but just for you to see the current status/quality:

• html (typedoc) https://cancerberosgx.github.io/demos/mirada-opencv-api-html/classes/_types_opencv_mat_.mat.html - using
• markdown (typedoc) https://github.com/cancerberoSgx/demos/blob/master/docs/mirada-opencv-api-md/classes/_types_opencv_mat_.mat.md

Although there are some details still, (mostly with references, enum types and formula rendering) I'm quite happy with it and now I'm it'm using it successfully to learn the library's API since I'm actually totally new with it) . Probably in editor-consumable documentation/type-declarations (which is my target) I should use shorter descriptions, but for now (and for me) I'm dumping the whole thing.)

I'm porting examples while I learn and use them library tests/ CI or in the playground and I must say the port of examples in general is quite straight forward (and surprisingly easier from C++) - I think I cover in most of the use cases shown in the tutorials. Nevertheless I will keep adding more as I learn.

As I said, I have suggestions for the js build that I will try to translate to PRs, right now I prefer to learn and enjoy a little bit more before that, and wanted to give this update just in case anybody has some opinions / feedback. I'm sorry for taking this issue mostly to discuss with myself before I have a PR proposal or a concrete contribution. Thanks

[frastlin]

This is really awesome, do you think you could do an example with SolvePnP and maybe Aruco markers?

It would also be really nice if we could use this in both the browser and React Native or Native Script.

[cancerberoSgx]

This is really awesome, do you think you could do an example with SolvePnP and maybe Aruco markers?

Sure if build supports them I will try. I recently attempted to add the module ml (machine learning) in the js build and although it is included by cmake it is not published in the global Module variable. I wonder if you know about any documentation about this topic since I'm kind lost here and would be awesome to try ml and other more training-oriented modules in the browser and node.js.

It would also be really nice if we could use this in both the browser and React Native or Native Script.

I'm running the tests in node.js - about those two technologies I don't really know them, but if they are oriented toward small devices (and performance/battery is important) I would use the c++/python versions not this one. Specially if you execute it as a command line since

Also since this project has so little JS implementation it would be interesting to try https://docs.assemblyscript.org/ or similar to generate also compile the browse clue code r and user's as webassembly altogheter. I will try to test things like https://github.com/as2d/as2d to see how/if this kind of things can be (ever used them) although I don't know if these things are related to React Native or Native Script - will let you know, thanks

[cancerberoSgx]

Is there a way to split the open.js file into modules so one can take advantage of tree-shaking?

Although I never tried, emcc has some support for dynamic linking, They say is still experimental and has some impact on the speed : * https://emscripten.org/docs/compiling/Building-Projects.html#dynamic-linking * https://github.com/emscripten-core/emscripten/wiki/Linking

• I have some experience tuning performanceI in the browser and int this case personally, I would not sacrifice speed just to reduce the output size. 1 MB more or less won't be noticed and there are other more efficient ways to alleviate this problem..

Then, emcc also supports this, which again I have no experience with: https://emscripten.org/docs/optimizing/Optimizing-Code.html#very-large-codebases ? what's your concern ? code size ? Do you think tree shaking you mention could improve also speed in this case? I have previous experience but only with WASM before, and here I suspect parsing such a big JavaScript is expensive for the browser. I tried without luck to build the .wasm I think should improve parsing at least, but I didn't had much time. Also I notice emcc uses only -O2 and -NO_exceptions by default. I wonder why not -O3 and if somebody tried the other optimization flags. Do you now if there is some benchmark that makes sense in JS to compare ?

Also a question, ImageMagick which targets a similar area, has two important compile flags that let you trade off between processing speed, memory and image output quality / match precision.
One is the quantum which is the depth for color channels and this affects memory consumption- n the past it used 8 bits but now several image formats support 16 and 32 . (now their default is 16) . The other flag is called HDRI that is related with Math and gives much better results in those filters / transformations involving complex arithmetic like like space domain transformations/ FFT, etc I'm not an expert but some operations needs that enabled or they won't work

The first flag impacts memory while this second flag impacts speed. just in case I want to know if there are any similar settings in opencv build ?

Thanks

[frastlin]

I wonder if you know about any documentation about this topic since I'm kind lost here and would be awesome to try ml and other more training-oriented modules in the browser and node.js.

There is an issue working through adding new modules to the build. That is about SolvePnP and some other calib modules, but if one is able to add those, then the ML modules shouldn't be that much different.

[peteruhnak]

Considering there hasn't been any change in the linked projects in ~15 months, I feel like it would be better to simplify the scope and just use what is already available, instead of trying to make everything perfect from the start.

Considering people do use the pre-built opencv.js directly, we could start with that, so the contents would be

• pre-built opencv.js (wasm variant presumably)
• package.json + tsconfig.json
• Promisified loader (to encapsulate the emscripten-specific onRuntimeInitialized)
• typescript bindings from JS: TypeScript type definitions #15306
• tests copied from the current code base/ JS: TypeScript type definitions #15306 (well, I rewrote them to TS to test the types & and fixed several issues in the code)

This setup doesn't really add anything extra (other than types which don't impact runtime or when using just JS).

Then at later time other things mentioned in the comments here can be addressed:

• initial slowdown / base64 encoding
• splitting the wasm file so it can be cached independently
• helpers.js cleanup/fixed imread/data copying in node
• optimization docs (e.g. I'm running the tests with --liftoff --no-wasm-tier-up which makes the loading ~order of magnitude faster; good for quick scripts, maybe not optimal for long-running jobs/workers/servers)
• whatever else mentioned here

---

So the current questions I have are:

• Is this setup sufficient?
• Should this be directly part of opencv/opencv, or maybe just the package contents in opencv/opencv-js (or opencv/opencv-ts)?
• What is the best way to cleanly load the lib both in node and browser?
  • for node in particular, I was implementing quite a lot of nonsense to avoid onRuntimeInitialized to be loaded multiple times ( https://github.com/peteruhnak/opencv-ts/blob/master/lib/loader.js ). It's been working so far, but looks really bad.

[frastlin]

I think you're right @peteruhnak if we could get something, that's better than nothing. What you have sounds good. I would say let's just put the JS files in a release, all the other files are just extra and not really needed. In the next releases, we can focus on Tree-Shaking and the other optimizations above.

[gabrielenosso]

Completely agree: we could cut development times making it available as it is.

[0xdevalias]

I don't think this was mentioned above already. I don't know much about it, but stumbled across this npm package + this issue while searching for a solution myself. Last release was 10 months ago, so may not be maintained, but might provide some solutions aligned to this thread:

• https://npm.io/package/@opencv.js/wasm
  • https://github.com/opencv-js/wasm

thanks:

• opencv/opencv for opencv.js
• https://github.com/echamudi/opencv-wasm for separateBinaryFile.js
• https://github.com/cancerberoSgx/mirada for types

---

https://github.com/echamudi/opencv-wasm seems to have been updated in January

Precompiled OpenCV to JavaScript + WebAssembly for node.js and deno environment. 🦕

In this Wasm-compiled OpenCV, there's no need to have OpenCV installed in the machine. The entire OpenCV library is already inside this package (opencv.js and opencv.wasm).

This module has zero dependencies.

Though echamudi/opencv-wasm#9 suggests that it's not currently optimised for a web environment.

[CezaryDanielNowak]

https://github.com/CezaryDanielNowak/fast-opencv-wasm
I updated a loader a bit so you can run OpenCV in the browser, Web Worker, and NodeJS. Wasm file is loaded as a binary instead of an inline base64 :)

[studentutu]

@CezaryDanielNowak sadly doesn't run in Electron ( neither main or render process, opencv is just not initialized) ( same as https://github.com/echamudi/opencv-wasm)