update SikuliX integration to interact by text, eg click some text.png

[kensoh]

No description provided.

[kensoh]

Fyi @siowyisheng on a draft PR. general test seems ok, probably the only thing left to improve is to see if can throw error for snap and read step if the text on screen to search for is not present.

Will ping here again when I think it's ready for your review and testing.

[kensoh]

@siowyisheng ready for your review, added showing error if text not found for snap and read.

[kensoh]

Adding on @siowyisheng, the implementation is straightforward but tedious as there is a need to update and create a lot of nested if else statements.

The strategy is if a .png or .bmp is given as the locator, and that image file does not exist on the computer, then it must be referring to interacting by text on the screen. Then route the action accordingly to interact by first creating a region to find that text on screen and passing it to be acted on according to the intent.

This feature is a great idea, thanks for suggesting it!

[kensoh]

PS - It also works for snap and read steps, and works for text with spaces.

[kensoh]

Adding a note call_sikuli() in tagui_parse.php and tagui_header.js needs updating to reflect this change. Same for present() and visible() in tagui_header.js. Because with the change, image file not found is no longer be an error as it gets interpreted as text OCR search. Will look into it.

[kensoh]

Making some draft changes above for appropriate error messages when text not found on screen, and the updated present() / visible() functions, plus visible_intent() in SikuliX tagui.py.

[kensoh]

@siowyisheng looks ready for your review and testing!

[siowyisheng]

It works for me, but I'm getting 2x Access is denied after my script. Also, I wonder if this is a good API. To write the .png on a file which is knowingly not there. I guess we should discuss a bit more?

START - automation started - Mon Feb 03 2020 16:17:54 GMT+0800 (Malay Peninsula Standard Time)

click C:/Users/ZBOOK15G5_46/home/projects/TagUI/Chlara.png

FINISH - automation finished - 9.8s

Access is denied.
Access is denied.

[kensoh]

Sure we discuss more! The 2 Access is denied is due to haven't merged the PR on hacking Chrome on Windows to exit cleanly. The PR that you've merge addresses that by first making sure that the file exist before overwriting and secondly to output any error to nul. There may be error due to the file missing or user has other open Chrome sessions, which prevents TagUI from writing to the Chrome preferences file.

For the writing of .png file, there is no creating of .png file. What happens is if there is no such file detected, the implementation will treat this as a text search instead of an image snapshot search base on the image file. This follows the consistent syntax of using .png / .bmp to denote that the identifier is using visual automation.

I couldn't think of other ways to invoke without either adding a new step or parameter to denote using text search, so I'm comfortable to implement this way, and also implemented this way to the RPA for Python project. I'm also maintaining a fork of TagUI v5.X here on my own - https://github.com/tebelorg/TagUI/tree/rpa_python

As the target user base is now different (TagUI for business user friendly), RPA for Python for Python programmers, I foresee divergence in how features are going to be implemented going forward, so I'll have to maintain a fork on my own just for the RPA for Python project.

Eg log files are first-class citizens there so I didn't implement the no logging by default in v6 for that fork, but only the text search and exit Chrome cleanly for Windows, out of the many v6 features. The change for consistency in echo / check / dump /write is also something that probably will break the package current implementation so I won't be rolling it out for the package in that fork.

[kensoh]

Following throws error, initial finding seems to be the call for get_text_for_sikuli($input_params) not generating proper JS syntax for if the parameter is a variable -

a = '1'
keyboard `a`[enter]

[kensoh]

Added 2 new commits above that fixes the issue. Below works now in normal run and live mode -

a = '123'
keyboard 123
keyboard `a`
keyboard [enter]
keyboard `a`[enter]
b = 'pc'
click pc.png
wait
click `b`.png

[kensoh]

Done, looks ok for me now. For your review and testing @siowyisheng

Syntax is step parameter(s) using ocr, the keyword using ocr is case-insensitive

[siowyisheng]

tested and LGTM, merging.

[kensoh]

Thanks @siowyisheng ! Found an issue when trying to use this with variable. Will compile into incorrect JS code and error out. Will figure out a solution and raise another PR 🤦‍♂