update() function to update package and TagUI files without internet [done]

[bobpan]

python test.py --proxy "https://proxy:8080"
[TAGUI][INFO] - setting up TagUI for use in your Python environment
[TAGUI][INFO] - downloading TagUI (~200MB) and unzipping to below folder...
[TAGUI][INFO] - /Users/bpan1
[TAGUI][ERROR] - failed downloading from https://github.com/tebelorg/Tump/releases/download/v1.0.0/TagUI_macOS.zip...
[Errno socket error] [Errno 8] nodename nor servname provided, or not known
[TAGUI][ERROR] - use init() before using url()
[TAGUI][ERROR] - use init() before using type()
[TAGUI][ERROR] - use init() before using read()

[kensoh]

I don't think python command has a --proxy switch. Initial googling seems to suggest that proxy setting is configured from operating system, and any internet traffic will be done through that.

Can you tell me more about your use case why are you using proxy? Is it because of firewall blocking on the company computer?

If yes, you can try using the pack() function on your personal laptop to generate a tagui_python.zip and tagui.py file. Which you can copy to your computer laptop to use with the usual import tagui as t. Details on pack() here - https://github.com/tebelorg/TagUI-Python#core-functions

There might be other dependencies though, which still requires the internet to install. I'm collecting inputs on such environments at this issue - #36 use in air-gap environments with network restriction and no internet)

[kensoh]

Adding on, inherently this tool does not have an option to connect to the internet by proxy. As the design for the tool is to emulate user's actions on the UI. So things that can be configured or done from the user interface on desktop, apps, webpage or command line can be done.

But things that are programmatically done and not doable by user are not in scope. That's also why I did not port over the API step from the original TagUI project that can be used to consume REST web services, but recommend users to use Python requests library to do that, when they call t.api().

[bobpan]

Can you tell me more about your use case why are you using proxy? Is it because of firewall blocking on the company computer?

Yes, our company needs to use a proxy to access the Internet. This exception occurred during the initialization phase. I disabled the code blocks and completed the installation manually, but I still want to use a proxy to install or update them.

# if not download (tagui_zip_url, home_directory + '/' + tagui_zip_file): return False

# if not download (tagui_delta_url, tagui_delta_file): return False

For test cases, there is no need for a network proxy

[kensoh]

I see.. If your company network can access GitHub through the proxy, you'll be able to get the updated TagUI files when you run the updated TagUI for Python tagui.py.

Alternatively, although user can manually copy the files listed in _tagui_delta() to replace the files on their computer, I think it is too much work to update version.

Let me think over what is a good way to reduce user friction. Maybe I can use similar idea in pack() to create a zip file of the updated files, so that user can copy the small zip to their company laptop easily. (using pack() works too now, but the file to copy is over 150 MB).

Feel free to add on and let me know what is the best way to copy files over to your computer laptop or any other constraints in your computer network. I think copy or email zip file directly will be the easiest way, but may be there are other interesting ways you use that is easier.

[kensoh]

Draft of a new function update(), it downloads the latest tagui.py for the package and the updated TagUI files, zip them up, convert the zip binary data into base64, generate an update.py containing code to update on target computer and embedded zipped data of the updated files in base64.

When python update.py is run on the target computer, the embedded zip data will be extracted and the respective files updated to the existing installation in the air-gapped environment without internet. Outstanding items before a new release can be published with this new function -

• to add code on getting the correct updated version number to stop sync on running
• move updated package file tagui.py to correct package folder from import tagui as t
• remove python-generated tagui.pyc and __pycache__/tagui.cpython-37.pyc (not required as python should recompile on detecting the file change in tagui.py)

With this new function, besides using proxy, users can simply copy over 1 file update.py to the restricted network computer to update to the latest version of TagUI for Python.

def update():
    """function to update package and TagUI files on an air-gapped computer without internet"""

    print('[TAGUI][INFO] - pack() is to deploy TagUI for Python to a computer without internet')
    print('[TAGUI][INFO] - update() is to update an existing installation deployed from pack()')
    print('[TAGUI][INFO] - downloading latest TagUI for Python and TagUI cutting edge files...')

    # first download updated files to tagui_update folder and zip them to tagui_update.zip
    if not os.path.isdir('tagui_update'): os.mkdir('tagui_update')
    if not os.path.isdir('tagui_update/tagui.sikuli'): os.mkdir('tagui_update/tagui.sikuli')

    tagui_python_url = 'https://raw.githubusercontent.com/tebelorg/TagUI-Python/master/tagui.py'
    if not download(tagui_python_url, 'tagui_update' + '/' + 'tagui.py'): return False

    # get version number of latest release for the package to use in generated update.py
    tagui_python_py = load('tagui_update' + '/' + 'tagui.py')
    v_front_marker = "__version__ = '"; v_back_marker = "'"
    tagui_python_py = tagui_python_py[tagui_python_py.find(v_front_marker) + len(v_front_marker):]
    tagui_python_py = tagui_python_py[:tagui_python_py.find(v_back_marker)]

    delta_list = ['tagui', 'tagui.cmd', 'end_processes', 'end_processes.cmd',
                    'tagui_header.js', 'tagui_parse.php', 'tagui.sikuli/tagui.py']

    for delta_file in delta_list:
        tagui_delta_url = 'https://raw.githubusercontent.com/tebelorg/Tump/master/TagUI-Python/' + delta_file
        tagui_delta_file = 'tagui_update' + '/' + delta_file
        if not download(tagui_delta_url, tagui_delta_file): return False

    import shutil
    shutil.make_archive('tagui_update', 'zip', 'tagui_update')

    # next define string variables for update.py header and footer to be used in next section
    # indentation formatting has to be removed below, else unwanted indentation added to file
    update_py_header = \
"""import tagui as t
import platform
import base64
import shutil
import os

tagui_update_zip = \\
"""

    update_py_footer = \
"""

# create update.zip from base64 data embedded in update.py
update_zip_file = open('update.zip','wb')
update_zip_file.write(base64.b64decode(tagui_update_zip))
update_zip_file.close()

# unzip update.zip to tagui folder in user home directory
if platform.system() == 'Windows':
    base_directory = os.environ['APPDATA'] + '/tagui'
else:
    base_directory = os.path.expanduser('~') + '/.tagui'
t.unzip('update.zip', base_directory + '/src')
if os.path.isfile('update.zip'): os.remove('update.zip')

# make sure execute permission is there for Linux / macOS
if platform.system() in ['Linux', 'Darwin']:
    os.system('chmod -R 755 ' + base_directory + '/src/tagui > /dev/null 2>&1')
    os.system('chmod -R 755 ' + base_directory + '/src/end_processes > /dev/null 2>&1')

# create marker file to skip syncing for current release
delta_done_file = t._py23_open(base_directory + '/' + 'tagui_python_' + __version__, 'w')
delta_done_file.write(t._py23_write('TagUI installation files used by TagUI for Python'))
delta_done_file.close()

# move updated package file tagui.py to package folder
shutil.move(base_directory + '/src/tagui.py', os.path.dirname(t.__file__) + '/tagui.py')
print('[TAGUI][INFO] - done. TagUI for Python updated to version ' + __version__)
"""

    # finally create update.py containing python code and zipped data of update in base64
    try:
        import base64
        dump("__version__ = '" + tagui_python_py + "'\n\n", 'update.py')
        write(update_py_header, 'update.py')
        update_zip_file = open('tagui_update.zip','rb')
        zip_base64_data = (base64.b64encode(update_zip_file.read())).decode('utf-8')
        update_zip_file.close()
        write('"""' + zip_base64_data + '"""', 'update.py')
        write(update_py_footer, 'update.py')

        # remove temporary folder and downloaded files, show result and usage message
        if os.path.isdir('tagui_update'): shutil.rmtree('tagui_update')
        if os.path.isfile('tagui_update.zip'): os.remove('tagui_update.zip')
        print('[TAGUI][INFO] - done. copy or email update.py to your target computer and run')
        print('[TAGUI][INFO] - python update.py to update TagUI for Python to version ' + tagui_python_py)
        return True

    except Exception as e:
        print('[TAGUI][ERROR] - ' + str(e))
        return False

[kensoh]

Ok implemented in v1.20! Available with pip install rpa --upgrade

To update on computers without access to internet -

1. import rpa as r; r.update() on computer with internet
2. copy or email update.py to target computer without internet
3. python update.py to update RPA for Python to latest version

This feature is still in beta (then in 2019), I'm open to hearing constraints faced by users so that I can improve the feature to work on their environments as best as possible. The goal of this feature is to make deployment as zero friction as possible on air-gapped computers without internet access.

There may be some OS-specific dependencies that need separate installation - for eg OpenSSL on macOS, MSVCR110.dll on Windows, or PHP on Ubuntu. If users can suggest ways to install them offline in their environments, I can also look into automating those setup.

Note - update() works by embedding binary zipped data of the updated files in base64 within the generated update.py (~100kb). update() can only be used if the target computer already has an existing installation of RPA for Python. If not, you can use pack() to generate the offline file (~150MB) for initial installation on your target computer without internet. More on pack() here.

[bobpan]

Thanks for @kensoh update, I think this would be a good solution without network access

[bobpan]

You can also update through a proxy in the following ways:

`
$ http_proxy="http://proxy.com:8080"

$ export http_proxy

$ sudo python test.py
`

[kensoh]

Thanks @bobpan for your feedback, and sharing on proxy!

I'll make a new issue referencing this solution, so other users can add on issues with proxy to work for a better solution like a t.proxy() function to set the proxy to use directly in script.

[4linuxfun]

I'm from China. Due to reasons that are well known, I can't access the github network effectively, so I used the manual download and replacement method of the file in tagui_delta (). After the operation is completed, the 'rpa_python' + version file needs to be created to check whether the operation is completed.

[kensoh]

@4linuxfun what you said is correct and works.

The pack() and update() functions depends on a user computer being able to access automated downloads from github.com domain, where the TagUI files are hosted. If the domain can't be accessed by the package, your hack above will work to update package and do initial installation. For initial installation, may also need to hack setup() function.

Is it that in China, for a normal user home laptop without company IT policy restriction, pack() and update() cannot work when it tries to download from github.com domain? I created pack() and update() functions assuming a user can use it from home laptop. Then copy the generated file over to work laptop with restricted internet access.

[4linuxfun]

@kensoh
I use the Proxifier tool proxy and can download the tagui package, but I cannot download the package under https://raw.githubusercontent.com/tebelorg/Tump/master/TagUI-Python/

[kensoh]

I see, maybe the way GitHub serves the URL (perhaps redirection) does not work with this proxy tool. Is there any web hosting sites that are free and accessible to China's normal internet?

If there is, you can consider downloading manually the required files, then host the files there. After that fork a version of this project and change all the https://raw.githubusercontent.com and https://github.com URLs to your hosted location. And perhaps publish a new package rpa_cn on pypi.org so users in China can use this package instead of the rpa package.

This tool is very stable and almost no significant change already past 6 months. So there will be no new releases that require you to keep updating your fork or rpa_cn Python package.