Dependencies packaged as manylinux1 wheels cannot be installed on Nix

[kierdavis]

[nix-shell:~/checkouts/j5api/j5]$ rm -rf ~/.cache/pypoetry/

[nix-shell:~/checkouts/j5api/j5]$ rm -rf pip-wheel-metadata

[nix-shell:~/checkouts/j5api/j5]$ poetry install
Creating virtualenv j5-py3.7 in /home/kier/.cache/pypoetry/virtualenvs
Installing dependencies from lock file


Package operations: 60 installs, 5 updates, 0 removals

  - Installing zipp (0.5.2)
  - Updating certifi (2018.11.29 -> 2019.6.16)
  - Installing entrypoints (0.3)
  - Installing importlib-metadata (0.18)
  - Installing markupsafe (1.1.1)
  - Installing mccabe (0.6.1)
  - Installing pycodestyle (2.5.0)
  - Installing pyflakes (2.1.1)
  - Updating pyparsing (2.3.1 -> 2.4.0)
  - Installing pytz (2019.1)
  - Updating urllib3 (1.24.1 -> 1.25.3)
  - Installing alabaster (0.7.12)
  - Installing atomicwrites (1.3.0)
  - Updating attrs (18.2.0 -> 19.1.0)
  - Installing babel (2.7.0)
  - Installing docutils (0.15.1)
  - Installing flake8 (3.7.8)
  - Installing imagesize (1.1.0)
  - Installing jinja2 (2.10.1)
  - Installing more-itertools (7.2.0)
  - Installing numpy (1.16.4)
  - Installing pluggy (0.12.0)
  - Installing py (1.8.0)
  - Installing pygments (2.4.2)
  - Updating requests (2.21.0 -> 2.22.0)
  - Installing snowballstemmer (1.9.0)
  - Installing sphinxcontrib-applehelp (1.0.1)
  - Installing sphinxcontrib-devhelp (1.0.1)
  - Installing sphinxcontrib-htmlhelp (1.0.2)
  - Installing sphinxcontrib-jsmath (1.0.1)
  - Installing sphinxcontrib-qthelp (1.0.2)
  - Installing sphinxcontrib-serializinghtml (1.1.3)
  - Installing toml (0.10.0)
  - Installing typing (3.7.4)
  - Installing wcwidth (0.1.7)
  - Installing cached-property (1.5.1)
  - Installing coordinates (0.3.0)
  - Installing coverage (4.5.3)
  - Installing fastcache (1.1.0)
  - Installing flake8-polyfill (1.0.2)
  - Installing isort (4.3.21)
  - Installing mypy-extensions (0.4.1)
  - Installing opencv-contrib-python-headless (4.1.0.25)
                                                                                                                     
[EnvCommandError]                                                                                     
Command ['/home/kier/.cache/pypoetry/virtualenvs/j5-py3.7/bin/pip', 'install', '--no-deps', 'opencv-contrib-pytho        
n-headless==4.1.0.25'] errored with the following output:                                                          
Collecting opencv-contrib-python-headless==4.1.0.25                                                                
  Could not find a version that satisfies the requirement opencv-contrib-python-headless==4.1.0.25 (from versions  
: )                                                                                                                
No matching distribution found for opencv-contrib-python-headless==4.1.0.25                                        
You are using pip version 19.0.3, however version 19.2.1 is available.                                             
You should consider upgrading via the 'pip install --upgrade pip' command.                                         
                                                                                                                     
install [--no-dev] [--dry-run] [-E|--extras EXTRAS] [--develop DEVELOP]

[trickeydan]

@RealOrangeOne This looks like it might be a zoloto issue. Any ideas?

[RealOrangeOne]

Absolutely none. The version definitely exists on PyPI.

@kierdavis is the package installable through pip directly? Or outside a nix-shell environment?

[trickeydan]

This might be an upstream bug in poetry.

https://circleci.com/workflow-run/899e500a-d9c9-4075-a550-9bfc64210b1d

[trickeydan]

We're not the only ones python-poetry/poetry#1256

[kierdavis]

The package's FAQ on PyPI suggests the problem may be due to an outdated pip, but I still experience the issue with the latest version available (19.2.1).

[RealOrangeOne]

Interesting. Looks like poetry might be swallowing errors or something, which is definitely strange. Have to monitor this one.

Installation works fine for me, so I assume it's something either non-determinstic, or environment-related.

[trickeydan]

Temporary fix for docutils applied in 9172f2f

[trickeydan]

It seems to be a conflict between poetry and pip 19.2

[kierdavis]

Same problem occurs using pip directly, so I don't think it's a poetry issue (EDIT: to be specific, there may be multiple causes of this symptom, and not all of them involve poetry):

[kier@saelli:~]$ nix-shell -p python3Packages.pip

[nix-shell:~]$ pip install --target=/tmp/throwaway-installation-dir opencv-contrib-python-headless==4.1.0.25 
Collecting opencv-contrib-python-headless==4.1.0.25
  Could not find a version that satisfies the requirement opencv-contrib-python-headless==4.1.0.25 (from versions: )
No matching distribution found for opencv-contrib-python-headless==4.1.0.25
You are using pip version 19.0.3, however version 19.2.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

As above, but with --verbose: https://gist.github.com/kierdavis/58e43264ecb493f7c88d1c328cafc7c9. It looks all available wheels are considered incompatible. I don't know enough about python packaging to offer any insight into why that is happening.

[RealOrangeOne]

it is not compatible with this Python

Well that's certainly different, but you're right that's definitely not a poetry issue! Nor is it a j5 nor zoloto issue. I fear the solution here is to 'wait and hope for the best', which kinda sucks!

[trickeydan]

We could do something similar to 9172f2f and pin opencv-contrib-headless to not that version

[RealOrangeOne]

Perhaps. @kierdavis Can you try installing some older versions, and see which is the most recent version you can install successfully. I'd be down for downgrading a couple versions if it helps make things more stable!

https://pypi.org/project/opencv-contrib-python-headless/#history

[kierdavis]

pin opencv-contrib-headless to not that version

Can you try installing some older versions, and see which is the most recent version you can install successfully

Unfortunately it appears this problem occurs independently of the version of opencv-contrib-python-headless:

[nix-shell:~]$ pip install --target=/tmp/throwaway-installation-dir opencv-contrib-python-headless 
Collecting opencv-contrib-python-headless
  Could not find a version that satisfies the requirement opencv-contrib-python-headless (from versions: )
No matching distribution found for opencv-contrib-python-headless
You are using pip version 19.0.3, however version 19.2.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

I also tried a few specific versions from the link @RealOrangeOne shared, no change.

[kierdavis]

pip seems to work fine for other packages:

[nix-shell:~]$ pip install --target=/tmp/throwaway-installation-dir beautifulsoup4
Collecting beautifulsoup4
  Downloading https://files.pythonhosted.org/packages/1a/b7/34eec2fe5a49718944e215fde81288eec1fa04638aa3fb57c1c6cd0f98c3/beautifulsoup4-4.8.0-py3-none-any.whl (97kB)
    100% |████████████████████████████████| 102kB 4.4MB/s 
Collecting soupsieve>=1.2 (from beautifulsoup4)
  Downloading https://files.pythonhosted.org/packages/35/e3/25079e8911085ab76a6f2facae0771078260c930216ab0b0c44dc5c9bf31/soupsieve-1.9.2-py2.py3-none-any.whl
Installing collected packages: soupsieve, beautifulsoup4
Successfully installed beautifulsoup4-4.8.0 soupsieve-1.9.2
You are using pip version 19.0.3, however version 19.2.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

[kierdavis]

tl;dr: the available wheels and my system use incompatible versions of manylinux

After doing some digging I've found that the Skipping link ...; it is not compatible with this Python is produced when the wheel is not compatible with the host system; specifically, when none of the built distribution compatibility tags provided by the wheel are supported by the host system.

From the verbose log posted earlier I identified a wheel that I'd expect to be considered compatible with my system (opencv_contrib_python_headless-4.1.0.25-cp37-cp37m-manylinux1_x86_64.whl: CPython 3.7, cp37m ABI, for 64-bit manylinux platform) and investigated what's going on with the compatibility tags.

tags supported by my system:
  ('cp37', 'cp37m', 'manylinux2010_x86_64')
  ('cp37', 'cp37m', 'linux_x86_64')
  ('cp37', 'abi3', 'manylinux2010_x86_64')
  ('cp37', 'abi3', 'linux_x86_64')
  ('cp37', 'none', 'manylinux2010_x86_64')
  ('cp37', 'none', 'linux_x86_64')
  ('cp36', 'abi3', 'manylinux2010_x86_64')
  ('cp36', 'abi3', 'linux_x86_64')
  ('cp35', 'abi3', 'manylinux2010_x86_64')
  ('cp35', 'abi3', 'linux_x86_64')
  ('cp34', 'abi3', 'manylinux2010_x86_64')
  ('cp34', 'abi3', 'linux_x86_64')
  ('cp33', 'abi3', 'manylinux2010_x86_64')
  ('cp33', 'abi3', 'linux_x86_64')
  ('cp32', 'abi3', 'manylinux2010_x86_64')
  ('cp32', 'abi3', 'linux_x86_64')
  ('py3', 'none', 'manylinux2010_x86_64')
  ('py3', 'none', 'linux_x86_64')
  ('cp37', 'none', 'any')
  ('cp3', 'none', 'any')
  ('py37', 'none', 'any')
  ('py3', 'none', 'any')
  ('py36', 'none', 'any')
  ('py35', 'none', 'any')
  ('py34', 'none', 'any')
  ('py33', 'none', 'any')
  ('py32', 'none', 'any')
  ('py31', 'none', 'any')
  ('py30', 'none', 'any')

tags provided by this wheel:
  ('cp37', 'cp37m', 'manylinux1_x86_64')

As it turns out there's two (presumably incompatible) versions of manylinux. It appears my system only supports manylinux 2010, but opencv-contrib-python-headless wheels are only provided for the older manylinux 1.

[kierdavis]

Can confirm that if I convince pip that my platform is manylinux1 instead of manylinux2010 then it finds and installs the package as expected. This is sort of a hack and I haven't checked if the package will actually work if installed in this manner, so I don't intend to use this as a solution.

[nix-shell:~]$ pip install --only-binary=:all: --platform=manylinux1_x86_64 --target=/tmp/throwaway-installation-dir opencv-contrib-python-headless==4.0.1.24 
Collecting opencv-contrib-python-headless==4.0.1.24
  Downloading https://files.pythonhosted.org/packages/c3/9e/6579e53e8eed1d09f6c4cc3bc55413b180de73a8d2d8e9043cb3dea4941b/opencv_contrib_python_headless-4.0.1.24-cp37-cp37m-manylinux1_x86_64.whl (25.2MB)
    100% |████████████████████████████████| 25.2MB 2.1MB/s 
Collecting numpy>=1.14.5 (from opencv-contrib-python-headless==4.0.1.24)
  Downloading https://files.pythonhosted.org/packages/fc/d1/45be1144b03b6b1e24f9a924f23f66b4ad030d834ad31fb9e5581bd328af/numpy-1.16.4-cp37-cp37m-manylinux1_x86_64.whl (17.3MB)
    100% |████████████████████████████████| 17.3MB 2.7MB/s 
Installing collected packages: numpy, opencv-contrib-python-headless
Successfully installed numpy-1.16.4 opencv-contrib-python-headless-4.0.1.24
Target directory /tmp/throwaway-installation-dir/bin already exists. Specify --upgrade to force replacement.
You are using pip version 19.0.3, however version 19.2.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

I'm pretty confident this isn't a j5-specific issue anymore, so I'll close this issue.

[kierdavis]

Nix is not manylinux1-compatible.

@trickeydan do you think this is a relevant portability issue? AFAIK we have no plans to deploy j5 on Nix, this is just for my personal development purposes. However we've already set a precedent by maintaining support for j5 on Windows.

Also, support for manylinux1 on Nix is in development, and based on this work I've made a start on modifying the Nix environment definition (shell.nix) to provide manylinux1 compatibility and allow this problematic package to be installed in my nix-manylinux1 branch. I'm still not sure whether this should eventually make its way into mainline j5 or if its too development-environment-specific.

[RealOrangeOne]

The fact there's currently a nix configuration in the repo makes it look like it's a supported platform. My suggestion is that either said file is fixed, or it's removed.

Personally i'd consider it fine to keep support for Nix, once NixOS/nixpkgs#55812 ships. What to do now I don't know. Windows support feels reasonable, as it's a common platform, and will be very useful when a simulator is created.

[trickeydan]

I see no reason why not, but it's a low priority.

[brano543]

@kierdavis How did you get this working in your environment? When I try to run shell.nix (https://github.com/j5api/j5/blob/nix-manylinux1/shell.nix) I am getting dozen of errors like "GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference".

Solving this problem would help me on RHEL7 when there is no possibility to use this solution https://sid-kap.github.io/posts/2018-03-08-nix-pipenv.html due to old kernel.

[kierdavis]

How did you get this working in your environment?

I haven't, yet. I'm expecting to see errors like those once I get round to finishing this (it's low priority for me at the moment).

[kierdavis]

NixOS/nixpkgs#71935 was merged, although I'm still experiencing this issue. Will need to investigate further.

[trickeydan]

Branch removed for cleanup. Changes on branch were:

{
  pkgsSrc ? <nixpkgs>,
  pkgs ? import pkgsSrc {},
}:

with pkgs;

let
  # Some of our dependencies are distributed as manylinux1 wheels, but Nix's
  # standard Python environment is not compatible with manylinux1
  # (https://github.com/NixOS/nixpkgs/issues/18484).
  # We can work around this by ensuring the shared libraries that must be
  # provided by the host system as part of the manylinux1 protocol (listed in
  # PEP 513) are made available on LD_LIBRARY_PATH.
  # This is based on the work done in https://github.com/NixOS/nixpkgs/pull/55812.
  manylinux1_lib_map = {
    "libpanelw.so.5" = ncurses5;
    "libncursesw.so.5" = ncurses5;
    "libgcc_s.so.1" = glibc;
    "libstdc++.so.6" = gcc7.cc;
    "libm.so.6" = glibc;
    "libdl.so.2" = glibc;
    "librt.so.1" = glibc;
    "libcrypt.so.1" = glibc;
    "libc.so.6" = glibc;
    "libnsl.so.1" = glibc;
    "libutil.so.1" = glibc;
    "libpthread.so.0" = glibc;
    "libresolv.so.2" = glibc;
    "libX11.so.6" = xorg.libX11;
    "libXext.so.6" = xorg.libXext;
    "libXrender.so.1" = xorg.libXrender;
    "libICE.so.6" = xorg.libICE;
    "libSM.so.6" = xorg.libSM;
    "libGL.so.1" = libGL;
    "libgobject-2.0.so.0" = glib;
    "libgthread-2.0.so.0" = glib;
    "libglib-2.0.so.0" = glib;
  };
  manylinux1_libs = runCommand "manylinux1-libs" {} ''
    mkdir -p $out/lib
    ${lib.concatStringsSep "\n"
      (lib.mapAttrsToList
        (name: pkg: "ln -s ${lib.getOutput "lib" pkg}/lib/${name} $out/lib/${name}")
        manylinux1_lib_map)}
  '';

in stdenv.mkDerivation {
  name = "j5-dev-env";
  buildInputs = [
    gnumake
    graphviz  # for docs
    python3
    python3Packages.poetry
  ];
  LD_LIBRARY_PATH = lib.makeLibraryPath [ manylinux1_libs libusb1 ];
}

[kierdavis]

closing due to inactivity; will reopen if this becomes a relevant problem again.