Expand usage of OpenCV.js-based modules

[jywarren]

Let's start by attempting to include OpenCV.js and do some simple operation with it:

https://docs.opencv.org/3.4.1/dd/d4d/tutorial_js_image_arithmetics.html

Thanks @abTest-123 -- https://publiclab.org/notes/amirberAgain/01-12-2018/python-and-opencv-to-analyze-microscope-slide-images-of-airborne-particles#c19541

[MargaretAN9]

@warren

I used the SciPy library for the sobel filter in the microscope video
https://publiclab.org/notes/MaggPi/08-14-2018/computer-vision-observations-of-elodea-plant-cells
I used this example because it showed how to merge the x and y filters.

http://www.turbare.net/transl/scipy-lecture-notes/advanced/image_processing/auto_examples/plot_find_edges.html
I realize now I could have probably done this with opencv , but the example never showed how to combine x and y.
The opencv sobel filter is below (In python)
https://docs.opencv.org/3.0-beta/doc/py_tutorials/py_imgproc/py_gradients/py_gradients.html

Not sure which one is fastest

[jywarren]

oh cool!

[jywarren]

(moving these comments on interest point matching to #300!)

[abTest-123]

I'm not really in the loop on this, but want to make sure you are familiar with the basic OpenCV tutorials, here is the feature matching example: https://docs.opencv.org/3.3.0/dc/dc3/tutorial_py_matcher.html Amir

…

On Tue, Sep 11, 2018 at 11:42 PM Jeffrey Warren ***@***.***> wrote: My ideal use would be: 1. run SURF on Image 1 to find interest points 2. run SURF on Image 2 to find interest points 3. find best matches of common points from each 4. in relation to image 1, rotate, scale, and translate image 2 to best overlap the top 2-3 common points between the 2 images 5. flatten into 1 image — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#237 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AjMD8nLUjwPHWt7zDrJprW_sxfnx3M7Xks5uaCAggaJpZM4TxTQF> .

-- Amir Bernat http://physweb.bgu.ac.il/~bernat/

[jywarren]

oh thanks!

…

On Wed, Sep 12, 2018 at 3:59 AM abTest-123 ***@***.***> wrote: I'm not really in the loop on this, but want to make sure you are familiar with the basic OpenCV tutorials, here is the feature matching example: https://docs.opencv.org/3.3.0/dc/dc3/tutorial_py_matcher.html Amir On Tue, Sep 11, 2018 at 11:42 PM Jeffrey Warren ***@***.***> wrote: > My ideal use would be: > > 1. run SURF on Image 1 to find interest points > 2. run SURF on Image 2 to find interest points > 3. find best matches of common points from each > 4. in relation to image 1, rotate, scale, and translate image 2 to > best overlap the top 2-3 common points between the 2 images > 5. flatten into 1 image > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > < #237 (comment) >, > or mute the thread > < https://github.com/notifications/unsubscribe-auth/AjMD8nLUjwPHWt7zDrJprW_sxfnx3M7Xks5uaCAggaJpZM4TxTQF > > . > -- Amir Bernat http://physweb.bgu.ac.il/~bernat/ — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#237 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJwHjhv4aX94nMd8lTsjWhmiCswk-ks5uaL6igaJpZM4TxTQF> .

[Mridul97]

Hey, I used OpenCV.js to add two images. It took two images as input. The tulips image below was given twice as both the inputs. Please have a look! 😃

And I got this image as output.

[jywarren]

oh wow. can you share a PR to see how you did this? Was it integrated into image sequencer, or just a demo? If not integrated, maybe you could share on gist.github.com so we have some code examples to go from? very cool!!!!

…

On Tue, Nov 13, 2018 at 10:01 AM Mridul97 ***@***.***> wrote: Hey, I used OpenCV.js to add two images. It took two images as input. The tulips image below was given twice as both the inputs. Please have a look! 😃 [image: tulips] <https://user-images.githubusercontent.com/21158457/48421586-d29bf100-e782-11e8-9132-917d54fadfa3.png> And I got this output. [image: output] <https://user-images.githubusercontent.com/21158457/48421634-e9dade80-e782-11e8-8bb7-92eb5bad3765.jpg> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#237 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJyd-TpMmE9CFiMVihJ17rtCNsxZgks5uut6lgaJpZM4TxTQF> .

[Mridul97]

No, It is not integrated into image sequencer. It is just a demo. Link of the gist :- https://gist.github.com/Mridul97/8588a6029b54942cd319605ad7a3917a
Please have a look! 😃

[jywarren]

Wow, but this is pretty simple (i simplified it a bit more even, and we'll have to watch asynchronous events):

mat1 = cv.imread(imgElement1);
mat2 = cv.imread(imgElement2);

let dst = new cv.Mat();
let mask = new cv.Mat();
let dtype = -1;
cv.add(mat1, mat2, dst, mask, dtype);
cv.imshow('canvasOutput1', dst);
mat1.delete(); 
mat2.delete(); 
dst.delete(); 
mask.delete();

Where is the cv variable defined?

In any case, this seems like it could be a pretty easy module to implement. But does it have to output to a canvas? canvasOutput1? Could we have it output an image object, or ImageData, or a data-url?

[jywarren]

Looking at https://docs.opencv.org/3.4.1/df/d24/tutorial_js_image_display.html ... maybe imshow is not canvas-specific?

[jywarren]

I saw there's a matFromImageData function, and am looking for a matching matToImageData function...

[jywarren]

You might also inspect what cv.imshow does in the source... i couldn't find this. Or inspect what's inside of mat1 and cv to see if we can parse it out ourselves?

[jywarren]

Oh awesome. So mat1.data is a Uint8Array(409600) -- we can work with that!

Maybe some help here:

• https://www.npmjs.com/package/array-to-image ??
• https://stackoverflow.com/questions/13385335/setting-uint8clampedarray-to-imagedata-is-very-slow-in-firefox
• https://stackoverflow.com/questions/50620821/uint8array-to-image-in-javascript#50621435
• https://www.html5rocks.com/en/tutorials/webgl/typed_arrays/

This is cool!

Note that opencv.js weighed 11 mb... whoa. But still potentially worth it!!!

[Mridul97]

Yes, we can access and modify pixel values using mat.data!! Something like this:

let row = 3, col = 4;
let src = cv.imread("canvasInput");
if (src.isContinuous()) {
    let R = src.data[row * src.cols * src.channels() + col * src.channels()];
    let G = src.data[row * src.cols * src.channels() + col * src.channels() + 1];
    let B = src.data[row * src.cols * src.channels() + col * src.channels() + 2];
    let A = src.data[row * src.cols * src.channels() + col * src.channels() + 3];
}

cv is actually defined in OpenCV.js.

We can also convert it to Uint8Array:

let arr = new Uint8ClampedArray(dst.data, dst.cols, dst.rows);

[MargaretAN9]

@jywarren Is openv.js installed on image sequencer? if so what version? If not, has anyway tried to install such as
https://docs.opencv.org/3.4/d4/da1/tutorial_js_setup.html ??

After install, need to determine if capture video and stitcher commands will work with public lab cameras.....

[jywarren]

it is not yet, but we'd love it to be!

…

On Wed, Mar 27, 2019 at 1:34 AM MargaretAN9 ***@***.***> wrote: @jywarren <https://github.com/jywarren> Is openv.js installed on image sequencer? if so what version? If not, has anyway tried to install such as https://docs.opencv.org/3.4/d4/da1/tutorial_js_setup.html ?? After install, need to determine if capture video and stitcher commands will work with public lab cameras..... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#237 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ1wnh3M5k082nl7NS-oosntM7TJJks5vawKygaJpZM4TxTQF> .

[jywarren]

OpenCV is installed now! See https://github.com/publiclab/image-sequencer/blob/main/src/modules/BlobAnalysis/BlobAnalysis.js for examples. Let's expand out from this initial example!

[jywarren]

Noting for context!!!

one issue I think was that there are pretty few WASM compilations available. we had to use an older version of OpenCV that someone had WASM compiled

This was all achieved by @aashna27 in #1185

[jywarren]

Uhhh, strangely it's not in the demo or the beta: https://sequencer.publiclab.org/examples/ https://beta.sequencer.publiclab.org/examples/

Though it definitely works in Aashna's demo: https://aashna27.github.io/image-sequencer/examples/#steps=brightness{},brightness{},blob-analysis{}

@aashna27 are you able to take a look to see how it got removed? And, we should add a test like this one to protect it!

https://github.com/publiclab/image-sequencer/blob/89fb3585ac082c6bce3cd72640ec2be9f75df5f0/test/core/modules/colorbar.js

[jywarren]

Solved -- it was a cache issue -- and opened an issue to request a module test for it: #1587

Thanks!