[Question] Ability to recognize on scaling?

[shirshak55]

Sometime resolution changes (Like adjusting browser) but still structure usually remain same so big button can become small button. For small difference it shouldn't matter too much right?

Is there anyway we can get finder to work so small changes don't matter. so we can

 await mouse.move(straightTo(centerOf(screen.find("printButton.png"))));

So sometime lets say u are searching printButton.png in browser but the size has changed slightly . Is there easy way to find location?

Thanks

[s1hofmann]

Image search is carried out over multiple scales, so nut.js is able to cope with changes in resolution automatically.
In case you're running into problems, one could also slightly reduce the required match confidence via screen.config.confidence

[sidorares]

@s1hofmann afaik currently search is performed on array of scaled images at [1, 0.9, 0.8, 0.7, 0.6, 0.5] scale - see

nut.js/lib/provider/opencv/template-matching-finder.class.ts

Line 65 in c131440

private scaleSteps = [0.9, 0.8, 0.7, 0.6, 0.5];

another option would be using feature points and ORB descriptors instead of comparing each pixel from (potentially scaled) template, this is currently not implemented but might be a good addition

[s1hofmann]

@sidorares Correct, current search uses a cascade of scaled images. A while ago I started some experiments using feature descriptors + sliding windows, but never finished it.

Would be interesting to see a comparison in terms of accuracy and performance, though.

EDIT: As a sidenote, the prebuilt package of OpenCV currently used by nut.js does not include any features2d package, all my experiments have been carried out with a custom build of opencv4nodejs

[sidorares]

on a big template on my examples feature descriptors performance was much better but for good accuracy you'd need to do pixel image matching / diffing once again after points transform matrix is calculated

[shirshak55]

actually i did tried with multiple resolution but it was not detecting the save button.

its chrome print button in darkmode osx.

Example.

both image looks nearly same but when i try it doesn't work till it matches same size of chrome window from where i took screenshot

Is it due to 99% confidence required ( Sorry i have no idea how opencv works)

[sidorares]

99% seems like a way too accurate for your example. Also note that border image is different ( 100% of template pixels contribute to match score )

[sidorares]

@shirshak55 try to use just inner part of a button without a border as a template:

[shirshak55]

well i did the experiment . What is did was first i took the screenshot of books folder icon and used the following code. If the screenshot is exact it does match properly. But if i zoom using Cmd+Plus that increase folder size by some amount and it doesn't work.

  screen.config.resourceDirectory = path.join(__dirname, "/../src")
  let a = await screen.find("books.png").catch((e) => {})
  console.log(a)
  return

works on exact size

Doesn't works if there is scaling in monitor or some application

I don't know if its hard problem to solve? Because here we are relying on opencv.

[sidorares]

it only compares short number of scale steps ( [0.9, 0.8, 0.7, 0.6, 0.5] ) and if you are unlucky and your real scale difference is in between of attempted steps accuracy is going to be quite low. Also current matching method does not tolerate rotations.

Try to set match accuracy to some low value ( 50% ? ) and check what is lowest one you still have correct find

[s1hofmann]

@shirshak55 Which version of nut.js are you using?
Starting with 1.4.0 an error message should inform you about the highest found match in case the required confidence value was not met.

As a side note to the described matching problems:
When encountering problems with image matching due to e.g. bigger differences in scale, another approach would be to switch to keyboard input to confirm a button etc.

[shirshak55]

@s1hofmann yep keyboard input would be great if there was a good way to switch to browser automatically because i execute script from terminal? browser resizing scales will definitely not be one of these [0.9, 0.8, 0.7, 0.6, 0.5] .

I don't know what is best way to accomplish it but i just want to do this. I think i should look on how robotjs works internally which might give idea of switching to chrome automatically.

[s1hofmann]

In case you want to perform both browser and desktop automation, you might want to take a look at Sakuli, which combines both browser automation and desktop automation in a single framework.

I do have plans on functionality for working with other processes / applications, but nothing is implemented so far.
Would love to hear back from your tests.