Some ideas for TagUI development

[kensoh]

Placeholder issue on some ideas of what's next for TagUI -

NECESSARY FEATURES

• General - helpers, ocr, visual, sql, remove technical blocks, apache 2, etc..
• Reusability - integrations with begin/finish, compounding with sub-scripts
• Extensibility - custom global / local functions, custom global / local repository
• Business - roro / ahk / excel / access / erp, ide helper, custom datatable
• Dependencies - update casperjs v1.1.4-1, sikuli v1.1.2, slimerjs v1.0

GOOD TO HAVE LATER

• Robotics - physical automation flows using robotic arm and openmv camera
• Holy Grail - ai-enabled recorder that learns workflows by watching videos

[kensoh]

Support of begin and finish keyword for integrations is implemented with #82. It makes calling Python, R, JS, Sikuli, CLIs easier, especially if there is a large code block to be run and developer prefers to put that code as part of TagUI script and not as a separate file for the respective platform.

[kensoh]

Support for calling other TagUI scripts within a script is implemented with #80 (more details there). With that commit, users can create reusable TagUI scripts that can be called from other scripts. Scripts can be compounded to do something much more complicated, while keeping flow readable and reusable.

[terrygoldman]

Great, thanks very much Ken for the great support and prompt updates.

I will give this a try and get back to you.

BR,
Terry

[kensoh]

Trying out a new Chrome extension improvement. A window popups immediately once user clicks on the note down element menu. This can help make it easier to check elements and write automation flows.

• to try this version on Chrome, go to the address chrome://extensions/
• download and unzip chrome.zip, then drag chrome folder into webpage
• disable old v3.1 extension, while recording right-click to note down element

UPDATE
Published the extension to Chrome web store. Added a new right-click option to inspect element. This will show up a popup of the element identifier so that users who are inspecting and writing their automation flows manually can see the ID without waiting for the recording to complete. More details here #89.

[kensoh]

A new ask step is added to allow interactive input from user. If run in default invisible or headless mode, the prompt will be shown in the shell window. If run in chrome mode, it will be a popup box.

The parameter for ask step is the question / prompt / instruction you have for the user. This feature is now usable from the cutting edge version. More details in the issue - #127

Response will be stored in ask_result variable.

[kensoh]

Added provision for double-click and right-click actions - details here #78

(works in visible and invisible Chrome, and visual recognition automation)

The step names are rtap / rclick and dtap / dclick

[kensoh]

Added support for user-defined global custom functions that persist with each TagUI releases. TagUI will check the tagui/src folder for a tagui_global.js file for user-defined functions.

These user-defined JavaScript functions will be accessible from all flow scripts. This feature is now usable from the cutting edge version. For more details, see issue #137

[kensoh]

Added support for a global repository file tagui_global.csv (place in tagui/src folder) that will make definitions available to all automation flows. This user-created file will persist with new versions of TagUI as this file is not included as part of the repository.

When there is the same definitions, flow-specific repository definitions take precedence over the global definitions. This global repo file works with a flow has datatable. More details at issue #139

[kensoh]

Added support for a local repository file tagui_local.csv that will make definitions available to automation flows in the same folder as the tagui_local.csv. This user-created file will persist with new versions of TagUI as this file is not included as part of the repository.

When there is the same definitions, flow-specific repository definitions take precedence over the local definitions and then over the global definitions. This local repo file works with a flow has datatable. More details at issue #139

This feature is now usable from the cutting edge version.

[kensoh]

Added support for local functions file tagui_local.js that make user-defined functions available to automation flows in the same folder as the tagui_local.js. This user-created file will persist with new versions of TagUI as this file is not included as part of the repository.

Flow-specific functions take precedence over the local functions and then over the global functions. More details at issue #137. This feature is now usable from the cutting edge version.

[kensoh]

Implemented a big codebase change that fundamentally affects all TagUI steps. The change allows deterministic flow of execution for some niche scenarios so that the order of actions, including steps that does not involve interacting with elements (such as echo, js step), and complex use of nested for, if, else, else if, can be done in the exactly the same order synchronous versus asynchronously. That is important as RPA and process automation micro-steps almost always need to be in sequence.

This change is now usable from cutting edge version. Details of change - #150

Breaks in backward compatibility in exchange for deterministic automation execution

1. var now strictly not supported for variables definition to be in scope. previously for some steps (eg echo, api), var variables will still be in scope
2. frame and popup after one another requires { and } steps. in general conditions (for, if else, else if, while) and popup / frame need { and } steps if used consecutively to prevent ambiguity
3. random use of { and } blocks without purpose no longer works. for example, when { and } steps are not used with if, else, else if, for, while popup, frame

Backward compatibility point 3 is now addressed by automatically supporting random code blocks by users that are not related to conditions or loops or frame or popup steps. I can't really think of a reason why a user would want to do that, perhaps for organizing sections of steps or keeping things readable. Nevertheless, updated codebase to support such random use of { and } to maximize backward compatbility.

For backward compatibility point 1, it is easy to update TagUI translation engine to always convert var variable_name = xxx to variable_name = xxx but this prevents advanced users who wants to limit scope of a variable within a section of JavaScript code from implementing good programming practices. Thus keep it flexible so that users can write variable_name = xxx according to TagUI documentation, or use var variable_name = xxx if they wish to limit scope for just that few consecutive JS statements.

For backward compatibility point 2, it is possible to write a pre-parsing algorithm in the translation engine to address that. However, that encourages bad scripting writing and ambiguity in code. Thus keeping it such that consecutive uses of conditions or loops or frame / popup steps require explicitly using { and } to define the code blocks in scope for such implementations. That promotes deterministic script-writing and easier maintenance as the script will show exactly what it is trying to do instead of leaving it to TagUI parsing engine to fill in the gaps.

[kensoh]

On a suggestion I heard recently - to run automation without restarting Chrome and closing user's browser tabs. This feature has been implemented and now usable from cutting edge release, which can be downloaded here to overwrite existing packaged installation. (do backup beforehand)

How it works is by creating a separate Chrome user profile for TagUI's use, so that profile is launched whenever TagUI automation is run. This way, existing Chrome browser tabs used by the user will not be closed and affect their work. More details at this issue.

Credentials (signed-in passwords and so on), will have to be re-entered as the browser will start off on a clean slate during the first run. Eg can use wait 30 seconds or live step to pause the automation to login.

[kensoh]

A few new features implemented into cutting edge release since above post.

See below and respective issues for more details -

• csv helper function csv_row() for easy handling of writing to csv files - CSV formatting enhancements #184
• extending present() and visible() to visual recognition using sikuli - Add Sikuli visual recognition for helper functions present() and visible() #202
• running complex statements for python and vision integration - Make lines between begin and finish for py, r, dom, run, vision steps run as a single block #167
• more csv improvements (csv_row() helper and custom datatable) - How to specify specific data table csv or object repository to be used? #175

Thanks @lohvht and @Aussiroth !

[kensoh]

Scrapped encrypting and auditing features, simply out-of-scope and controlled from OS level - more details here - #217

[kensoh]

Also, I am scrapping efforts on developing enterprise features such as a dashboard for centralized deployment, and a reporting interface for gathering execution details to be viewed centrally.

Commercial RPA software does this very well and I believe the niche for TagUI is in personal desktop automation in a distributed way. Use cases where benefit individual users but not creating that much financial value that the company want to dedicate 1 commercial RPA user license to the laptop.

Efforts on creating these enterprise features should rather be reserved to features such as making it easier for users to create script (eg the Italian guy's sample of auto-filling up the steps), or more pressing stuff such as integration with Windows desktop applications.

[kensoh]

Closing this issue for now,

As outstanding items on desktop apps is tracked by #113 and ide helper is tracked via #188.

Hardware robotics integrations and video computer vision automation can be raised later.

[kensoh]

I'm only looking at 3 items in pipeline for TagUI before hitting maintenance mode.

1. integrating with desktop apps - TagUI for Desktop Applications --> use visual automation #113
2. assistant for writing scripts - TagUI Writer 1.01 : helper tool for coding (most useful commands for beginners) #188
3. for loop break and continue - For loop bugfix - explore enabling break and continue within for loops #216

May reach out to other open-source RPA software maintainers to look at collaboration. Was thinking yesterday if can make a great open-source RPA tool and pass on to @microsoft or another large tech company to maintain, can put pressure on commercial RPA tools to raise the quality and ease-of-use of their free versions. That should lead to the largest impact on the RPA ecosystem.