Installing Python sources outside the site-packages

[igo95862]

I want to install the Python files outside the site-packages directories so that it is not Python version dependent.

My first idea was to modify the python.purelibdir and python.platlibdir options, however, it is impossible to programatically modify the options unless they are in subproject

I tried setting subdir to an absolute path but then it no longer compiles byte code.

Instead I found this trick:

py_installation.install_sources(
    source_files,
    subdir : fs.relative_to('/' / python_packages_dir, py_installation.get_path('purelib')),
)

You get both byte code compilation and it is installed outside the site-packages. However, it is kind of a hack.

Maybe there is a better solution?

[eli-schwartz]

Why do you want it to be independent of the python version?

Note that on e.g. Debian, the default installation location for install_sources is already "python3/dist-packages" which is independent of version

That's actually a bit of a problem since you can install the interpreter for python 3.8 and python 3.10 and then install a system package for a python 3.10 module, which is then importable in python 3.8 but is broken and fails to work (despite which, pip install thinks it's already installed and doesn't try to install a compatible copy).

You cannot actually use a module with multiple versions of python unless it's a pure library. For applications (such as bubbblejail??) you are limited to whichever version of python is in the script shebang -- and also whichever version of python has your python dependencies installed.

[igo95862]

What about non-Arch distros? :)

This setup also works for any other distro. (python_packages_dir = get_option('libdir') / 'bubblejail/python_packages/' ) For distro maintainers there is a build option to use the py_installation.get_path('purelib') which I know the Alpine Linux uses.

The other downside is that if you ever run bubblejail as root

You aren't supposed to run it as root as it is completely unprivileged. It expects to be run in the desktop environment as without XDG_RUNTIME_DIR it will fail to run. Sometimes people run it as root probably because of how firejail works so I should probably add the abort to launch script then it runs as root. There is also sys.dont_write_bytecode which can be manipulated to disable bytecode or the launch script shebang can include the -B to disable bytecode writing.

[eli-schwartz]

This setup also works for any other distro.

It's an unusual complication, I'd say, for any distro that officially packages bubblejail. In particular it makes it harder to detect cases where the package should be rebuilt (you can't just check for packages which install files to a deprecated directory).

You aren't supposed to run it as root as it is completely unprivileged.

Bytecode compilation occurs at import time, which also means when running the program as root in order to see the --help text.

[igo95862]

Bytecode compilation occurs at import time, which also means when running the program as root in order to see the --help text.

I can manipulate the sys.dont_write_bytecode before importing any bubblejail's modules.

Anyway this is a minor detail. The important part is that I want to make bubblejail survive Python upgrades.

If meson does not have a good way of installing the Python sources outside the purelib I will continue using my bytecode compiler. I also plan on using a subproject with my Python namespaces library. It looks like I manipulate the subproject settings to modify the platlibdir of the subproject. Not sure how the byte code compilation will work.

An alternative solution I can think of is recording the build time purelib in the launch script and then checking if it is still in the sys.path before importing bubblejail modules. If it is no longer there that means the Python version was changed so insert it back and maybe emit a warning that user should rebuild their bubblejail. Not sure how vendoring the subprojects will work as I don't want to install the vendored subprojects inside the purelib or they might conflict. Maybe record an extra sys.path in the launch script with the vendored modules path.

[igo95862]

I've upgraded to Python 3.12 and bubblejail still works without rebuilding meaning my approach of installing in to a version independent directory is validated.

[eli-schwartz]

• that doesn't solve the bytecode issue
• it definitely doesn't validate the approach for other software, which means I'm hesitant to provide a mechanism other software might be tempted to use also
  • it's not guaranteed to work for bubblejail either when upgrading to 3.13, just because it happened to work for 3.12