pkgsStatic.python3 has no builtin modules

[r-burns]

Describe the bug

pkgsStatic.python3's builtin modules fail to build (without causing the build as a whole to fail), and are unavailable for import in the built python3 package.

Steps To Reproduce

$ nix-shell -p pkgsStatic.python3
$ python3 -c "import readline"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'readline'

Additional context

Each module fails to build with these errors:

/nix/store/ckymfxng8g4xpnix8apbvyx03xdg9vdb-x86_64-unknown-linux-musl-binutils-2.35.1/bin/x86_64-unknown-linux-musl-ld: /nix/store/l05ny4v41irzq2ns3qk9615h4603pmbr-x86_64-unknown-linux-musl-stage-final-gcc-10.3.0/lib/gcc/x86_64-unknown-linux-musl/10.3.0/crtbeginT.o: relocation R_X86_64_32 against hidden symbol `__TMC_END__' can not be used when making a shared object
/nix/store/ckymfxng8g4xpnix8apbvyx03xdg9vdb-x86_64-unknown-linux-musl-binutils-2.35.1/bin/x86_64-unknown-linux-musl-ld: /nix/store/l05ny4v41irzq2ns3qk9615h4603pmbr-x86_64-unknown-linux-musl-stage-final-gcc-10.3.0/lib/gcc/x86_64-unknown-linux-musl/10.3.0/crtend.o: relocation R_X86_64_32 against `.ctors' can not be used when making a shared object; recompile with -fPIC
/nix/store/ckymfxng8g4xpnix8apbvyx03xdg9vdb-x86_64-unknown-linux-musl-binutils-2.35.1/bin/x86_64-unknown-linux-musl-ld: /nix/store/acv2cigbl2s428g1cd5zgs0v6hi34wgh-ncurses-6.2-x86_64-unknown-linux-musl/lib/libncursesw.a(lib_tparm.o): warning: relocation against `_nc_tparm_err' in read-only section `.text'
collect2: error: ld returned 1 exit status

Notify maintainers

@FRidh

Metadata

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.13.12, NixOS, 21.11 (Porcupine)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.15`
 - channels(root): `"nixos-21.11pre311431.88226ea038e"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: pkgsStatic.python3
# a list of nixos modules affected by the problem
module:

[r-burns]

So it's possible to build these statically into cpython itself, by putting 'static' at the top of Modules/Setup and uncommenting any needed modules.

However, there's still the issue of other loadable modules, which need to be dynamic libs for cpython to be able to load them, so pkgsStatic can have problems building them, e.g. python3.pkgs.libxml2. And trying to load these modules seems to fail because we are using musl's nonfunctional stub dlopen:

$ nix-shell -p pkgsStatic.python3 -p python3.pkgs.libxml2 --run 'python3 -c "import libxml2"'                                                                 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/nix/store/s65dqlhjji7x6m4girn1ifzg4z1jcgys-libxml2-2.9.12-py/lib/python3.9/site-packages/libxml2.py", line 1, in <module>
    import libxml2mod
ImportError: Dynamic loading not supported

So I still don't have a good solution here, ideally we'd be able to build a static cpython which can still dlopen loadable modules.

[KAction]

Or, alternatively, pkgsStatic.python.withPackages re-compiles python with more libraries. No idea how to implement it, though.

[r-burns]

I'm not sure that's a great solution, there are so many packages that depend on python that we'd have a huge amount of closure bloat if each one of these needs its own bespoke recompiled python with integrated libraries.

I think python is so tied to the concept of dynamically loading modules, that it wouldn't be surprising to have a python interpreter which is still loading dynamic modules in an otherwise static distribution. We'd still ensure that the python interpreter itself and its builtins are statically compiled, anyway.