Parallel execution controller for automation farms

[kensoh]

Referencing - https://github.com/tebelorg/TagUI/issues/42#issuecomment-320465044

Parallel runs are good to speed up something, provided that process is not tied with a single user account that is meant for single-login at any one time. TagUI already has web-service for Linux/macOS to allow triggering via URL call. This is a capability which deserves looking further into.

To deploy parallel runs, each machine needs to be a server node and a separate controller script can send the execution requirements to the webservice servers accordingly. Alternatively, each slave can connect to a single master controller machine to retrieve inputs for execution and feedback through the controller webservice. Key functionalities - distribute flows, execute, monitor status, collect results..

[kensoh]

Was chewing on this for a couple of days. Decided against implementing a parallel execution controller for now. The reason is intentionally pace out how much TagUI can scale.

That is also same reason why the datatable format is in the non-standard data-centric way versus the usual way which columns and rows are swapped from the TagUI datatable format. It takes between 1-9 lines in PHP to transpose an array of rows and columns. The reason for making the default format a non-mainstream format, and also not including a function to transpose is to limit the tool.

I have nothing against data-scrapping use cases or load-testing applications. But making it too easy to scale may end up attracting the wrong users with the wrong expectations of what the tool is designed for. Furthermore, there are established leading open-source tools specifcally for those purposes.

May reopen and revisit in future. At the moment no personal need to implement something like this.

[kensoh]

response to a user asking about parallel runs

Yeah TagUI was not designed to run in parallel, although separate folders on the same laptop computers it can run batch automation using separate csv files. Still it does not have a controller to scale the automation in parallel across multiple computers. I didn't add that part into its core features as parallel runs can be easily misused for bad and malicious purposes.