[use-case question] How to compile a package with subpackages, and use/import all its modules from python scripts?

[mjscosta]

---

Required Info
Nuitka version	0.5.29.3/pip
Operating System & Version	Ubuntu 16.04.4 LTS
Python version	Python 2.7.12

Hi Kay Hayen, first of all, congratulations for your outstanding work with nuitka, and project methodologies. I find that the documentation is somehow poor, which I'd like to volunteer to contribute with use-cases and examples explained in the documentation/how-tos. On that regard I have some questions about a use-case, that I cannot figure out how to use.

I have a package, with sub-packages inside the directory python_path_root (see below),
I would like to build a single library with nuitka that contains all the packages and sub-packages, and be able to import any of its modules from other python scripts. Is this possible?

Example:

#script1.py

from package1.subpackage1 import submodule11

submodule11.do_something()

#script2.py

from package1 import module1
from package1.subpackage1 import submodule11

submodule11.do_something()
module1.do_something_else()

Package structure

├── python_path_root
│   └── package1
│       ├── __init__.py
│       ├── module1.py
│       ├── module2.py
│       ├── packaging
│       │   ├── __init__.py
│       │   └── pyinstaller_specs.py
│       ├── subpackage1
│       │   ├── __init__.py
│       │   └── submodule11.py
│       ├── subpackage2
│       │   ├── __init__.py
│       │   └── submodule21.py

[kayhayen]

Did you check out "Use Case 3 - Package compilation" in the User Manual ? It should be exactly that what you are doing.

For factory branch I updated the manual a bit, because --recurse-directory is about file system, and we are moving to --include-package/module options, which will work with names of packages/modules instead. The use case you have works right now and is supposed to become even simpler.

Yours,
Kay

[mjscosta]

Thanks for your reply,
Actually I've omitted a detail, that is, one of the submodules of a subpackage is a swig module:

├── python_path_root
│   └── package1
│       ├── __init__.py
│       ├── module1.py
│       ├── module2.py
│       ├── packaging
│       │   ├── __init__.py
│       │   └── pyinstaller_specs.py
│       ├── subpackage1
│       │   ├── __init__.py
│       │   └── submodule11.py
│       ├── subpackage2
│       │   ├── __init__.py
│       │   └── submodule21.py
│       └── subpackage3
│           ├── __init__.py
│           └── swigpkg
│               ├── CMakeLists.txt
│               ├── __init__.py
│               ├── myswig_module.cpp
│               ├── myswig_module.h
│               ├── myswig_module.i
│               ├── myswig_module.py
│               ├── myswig_modulePYTHON_wrap.cxx
│               └── _myswig_module.so

I'll check the factory version for the new doc and updates.
Best,
Mario

[kayhayen]

Ah, that ".so" can of course not be included in the package. Compile with Nuitka as described, and produce a directory structure (yourself) like this:

package1.so
package1/swigpkg/_myswig_module.so

Nuitka will use the directory for loading stuff. It at least works for standalone mode, where it has to, and I think it will for package mode too. Please confirm to me if this works indeed.

[mjscosta]

Hi Kay,
I've tested with:

0.5.29.4 (from pip), and following the "Use Case 3", with nuitka --module package1 --recurse-directory=package1

I've got this warning, but it works properly, with and without swig.

Nuitka:WARNING:Recursed to package 'package1' at 'package1' twice.
Nuitka:WARNING:Not recursing to 'package1.subpackage1.submodule11' (/home/.../test_nuitka/output3/package1/subpackage1/submodule11.py), please specify --recurse-none (do not warn), --recurse-all (recurse to all), --recurse-not-to=package1.subpackage1.submodule11 (ignore it), --recurse-to=package1.subpackage1.submodule11 (recurse to it) to change.

factory branch (nuitka v 0.5.30rc4), the option --include-package/module does not seem to be present, and using nuitka --recurse-to=package1 --recurse-to=subpackage1 --module package1 does not seem to recurse into the subpackages.

I've setup a set o examples/tests to test nuitka, before applying to a real more complex scenario, I've also build a directory walker script that helps me building single modules recursively, cleaning build stuff and .py/.pyc and avoid compiling swig modules with nuitka.

You can checkout from here:
https://github.com/mjscosta/nuitka_testcases

Note that the test case numbering is not related with the use cases from nuitka manual.