Compile into single executable

[Eeems]

In order to save time on installing dependencies etc, it would be good to compile this into a single executable with something like pyinstaller or nuitka

[Eeems]

Did a simple test with nuitka, and it looks like you just need to add it ans scons to the requirements in order to get it building. There are some warnings, but it still builds and runs (at least with just the --help flag).

eeems@shark  ~/git/Github/toltec-dev/build   main  python -m nuitka toltecmk                                                               
Nuitka-Options:INFO: Used command line options: toltecmk
Nuitka:INFO: Starting Python compilation with Nuitka '0.6.16.2' on Python '3.9' commercial None.
Nuitka-Inclusion:WARNING: Not following import to 'toltec' (/home/eeems/git/Github/toltec-dev/build/toltec), please specify --nofollow-imports (do not warn), --follow-imports (recurse to all), --nofollow-import-to=toltec (ignore it), --follow-import-to=toltec (recurse to it) to change.

Nuitka-Inclusion:WARNING: Not following import to 'toltec.builder' (/home/eeems/git/Github/toltec-dev/build/toltec/builder.py), please specify --nofollow-imports (do not warn), --follow-imports (recurse to all), --nofollow-import-to=toltec.builder (ignore it), --follow-import-to=toltec.builder (recurse to it) to change.

Nuitka-Inclusion:WARNING: Not following import to 'toltec.recipe' (/home/eeems/git/Github/toltec-dev/build/toltec/recipe.py), please specify --nofollow-imports (do not warn), --follow-imports (recurse to all), --nofollow-import-to=toltec.recipe (ignore it), --follow-import-to=toltec.recipe (recurse to it) to change.                                                                                                                   

Nuitka-Inclusion:WARNING: Not following import to 'toltec.repo' (/home/eeems/git/Github/toltec-dev/build/toltec/repo.py), please specify --nofollow-imports (do not warn), --follow-imports (recurse to all), --nofollow-import-to=toltec.repo (ignore it), --follow-import-to=toltec.repo (recurse to it) to change.
                                                                                                                                                
Nuitka-Inclusion:WARNING: Not following import to 'toltec.util' (/home/eeems/git/Github/toltec-dev/build/toltec/util.py), please specify --nofollow-imports (do not warn), --follow-imports (recurse to all), --nofollow-import-to=toltec.util (ignore it), --follow-import-to=toltec.util (recurse to it) to change.

                                                                                                                                                Nuitka:INFO: Completed Python level compilation and optimization.                                                                               
Nuitka:INFO: Generating source code for C backend compiler.
                                                                                                                                                Nuitka:INFO: Running data composer tool for optimal constant value handling.                                                                    
Nuitka:INFO: Running C level backend compilation via Scons.                                                                                     
Nuitka-Scons:INFO: Backend C compiler: gcc (gcc).
Nuitka-Scons:WARNING: You are not using ccache.                                                                                                 
Nuitka:INFO: Keeping build directory 'toltecmk.build'.
Nuitka:INFO: Successfully created 'toltecmk.bin'.

Testing with --follow-imports now.

[Eeems]

Looks like it'll build perfectly happily with the following:
python -m nuitka --follow-imports --plugin-enable=pylint-warnings toltecmk. The output file is 12M in size.

I'll continue to test with --onefile

[Eeems]

With --onefile the resulting file is 17M and seems to run with --help fine as well. Here is what I ran to get it to work:

python -m venv env
source env/bin/activate
python -m pip install -r requirements.txt
python -m pip install scons
python -m pip install nuitka
python -m nuitka --follow-imports --plugin-enable=pylint-warnings --onefile --linux-onefile-icon=/path/to/python.xpm --assume-yes-for-downloads toltecmk

[matteodelabre]

Thanks, this is an interesting approach. We would need to make sure that it doesn’t conflict with #17 (can we still dynamically load and run Python code from the compiled binary?).

Is there a reason why turning this into a pypi module wouldn’t work for your use-case? Then install would simply be a matter of pip install toltecmk.

[Eeems]

Thanks, this is an interesting approach. We would need to make sure that it doesn’t conflict with #17 (can we still dynamically load and run Python code from the compiled binary?).

Ah, good point, I know with pyinstaller there are ways to make it happen. It might be possible with nuitka as well though.

Is there a reason why turning this into a pypi module wouldn’t work for your use-case? Then install would simply be a matter of pip install toltecmk.

This would also need to install all the dependencies, which would take longer than pulling down a 17M file. It would also require that anybody using this has python/pip installed.

[matteodelabre]

Another aspect I just thought about is that the pypi module can be useful to reuse only part of the code in other projects, such as the recipe-parsing code, or the ipk-reading-writing capabilities. Actually, we may want to leverage this when integrating this repo into the main repo’s build script.

Regarding the dependencies, note that the list in requirements.txt includes both production and development deps. The set of dependencies needed for only running toltecmk is a bit smaller:

certifi==2021.5.30
charset-normalizer==2.0.4
docker==5.0.0
idna==3.2
python-dateutil==2.8.2
requests==2.26.0
six==1.16.0
urllib3==1.26.6
websocket-client==1.1.0

That being said, I understand your use case and I guess there’s no reason why we couldn’t publish both a pypi module and a standalone binary. Although we may want to wait until #17 is resolved so that we can make sure it’s working with dynamic module loading first, if you don’t mind. I’m planning to work on this during this week anyway.

[matteodelabre]

I did some tests and Nuitka seems to work fine with dynamic imports and the hook system overall! A small caveat: as one may expect, modules imported as hooks from the standalone binary cannot import any module installed on the host, only those packaged in the binary (i.e., Python standard lib and Toltec modules).

[Eeems]

You can probably just add them to the path and they'll import. I'll try to remember to add some sample code where I did this with a pyinstaller project.

[Eeems]

Okay, to allow importing from the full standard lib, I just compress the entire lib into a zip file, and then you just need to do the following to ensure it'll prefer loading from the zip:

sys.path.insert(0, "lib.zip")

import http.cookies

This does have a caveat though, anything from the standard lib that is partially supplied by the nuitka compile (at least with pyinstaller this is the case) will need to be explicitly imported to get the full library:

def ImportFromLib(name: str) -> any:
    try:
        return zipimporter("lib.zip").load_module(name)

    except ZipImportError:
        pass
    except ImportError:
        pass

    raise ImportError("Unable to load {} from lib.zip".format(name))

http = ImportFromLib("http")
import http.cookies