python: add patchelf option

[bobvanderlinden]

Currently Python package managers install packages with prebuilt binaries, which might refer to libraries that the program is not able to find.

LD_LIBRARY_PATH is a solution, but seeps through to sub-processes, sometimes resulting in surprising errors due to conflicting libraries/interpreters.

I have been experimenting with using autopatchelf on venv. A tool that finds ELF files, looks up their dependencies in a set of lib directories and patches the ELF files to use those lib directories specifically. It seems to work fairly well.

There are a few downsides though:

• This mutates the files that are 'managed' by a different package manager, which could lead to confusion when using the package manager manually.
• It adds some initialization time (~8s for my project), mostly to lookup all .so* files in the venv.
• It doesn't solve the problem for libraries that are dynamically loaded without them being a requirements in the ELF header.
• It isn't a solution for OSX (yet?)

I mostly wanted to give this option some consideration compared to LD_LIBRARY_PATH. Feel free to decline this. I'm open to all feedback.

[Mic92]

I don't know the devenv code well enough but find this approach interesting.
Here a few hints: 1. Avoid running patchelf repeatably as might change the binary on every run (sometimes with weird side effects). 2. Make sure that ${config.devenv.profile}/lib cannot be garbage collected.

[bobvanderlinden]

Thanks for the feedback!

1. Avoid running patchelf repeatably as might change the binary on every run (sometimes with weird side effects).

Indeed. Maybe storing the profile inside VIRTUAL_ENV and comparing it is a good enough approach.

2. Make sure that ${config.devenv.profile}/lib cannot be garbage collected.

I presumed the profile is not GCed, as it has a GC root that in .devenv. Not entirely sure whether this is a good presumption though.

[Mic92]

macOS has install_name_tools instead of patchelf. Maybe just adding .devenv/lib with that might be good enough?

[Mic92]

I wonder if this should be a no-op on non-elf architectures to not break macOS by accident.

[Atry]

I found this PR is similar to mach-nix and dream2nix.

[domenkozar]

I think the method that we use by fixing the interpreter itself now works well in 1.0 branch, we're only waiting for the merge of NixOS/nixpkgs#275701

[nevesenin]

@bobvanderlinden Something like this might be helpfull for ruby too. I added this

enterShell = ''
  ${inputs.autopatchelf.packages."${system}".default}/bin/autopatchelf --bintools ${pkgs.bintools} --patchelf ${pkgs.patchelf}/bin/patchelf --libs $DEVENV_PROFILE/lib --path $BUNDLE_PATH
'';

to a RoR projects config. I didn't find another way to make the dartsass-rails gem work.

[bobvanderlinden]

The method used in NixOS/nixpkgs#275701 should also work for Ruby, but it currently has only been implemented for Python.

[nevesenin]

Oh, great. I found that PR before but lost track while reading through it.

[domenkozar]

Going to close this for now, we found a way to make virtualenv work well with Nix.

To solve glibc issues we'll have to use LD_AUDIT hack or something similar.

Let me know if there's any other reason we'd have this