Make fractal-tasks-core installable on Apple silicon macs

[jluethi]

Currently, when trying to install fractal-tasks-core on my M1 Pro MacBook Pro, I run into a few issues with dependencies. One is with Qt:

ERROR: Could not find a version that satisfies the requirement PyQt5-Qt5<6.0.0,>=5.15.2 (from fractal-tasks-core) (from versions: none)
ERROR: No matching distribution found for PyQt5-Qt5<6.0.0,>=5.15.2

=> see discussion in napari-segment-blobs-and-things-with-membranes to drop this dependency, such that we can drop it from fractal-tasks-core (I currently manually removed it from the pyproject.toml to be able to run my installation and test some of the parsing functionality)

Another library that is making issues is imagecodecs:

[...]
n file included from /Users/joel/opt/miniconda3/envs/metadata-parsing/lib/python3.9/site-packages/numpy/core/include/numpy/ndarraytypes.h:1948:
      /Users/joel/opt/miniconda3/envs/metadata-parsing/lib/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: "Using deprecated NumPy API, disable it with "          "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
      #warning "Using deprecated NumPy API, disable it with " \
       ^
      imagecodecs/_aec.c:756:10: fatal error: 'libaec.h' file not found
      #include "libaec.h"
               ^~~~~~~~~~
      1 warning and 1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

Installing with conda seems to work (conda install imagecodecs). But then our current pyproject.toml specifies at least the 2022.2.22 version, which isn't available on conda. Can we ease this requirement? Conda can install 2021.8.26 on my mac successfully, all the pip installations appear to fail.

[tcompa]

We don't use imagecodecs ourselves, but it is a dependency of cellpose - which does not require any specific version:

$ poetry show cellpose

 name         : cellpose                          
 version      : 2.1.1                             
 description  : anatomical segmentation algorithm 

dependencies
 - fastremap *
 - imagecodecs *
 - llvmlite *
 - natsort *
 - numba >=0.53.0
 - numpy >=1.20.0
 - opencv-python-headless *
 - scipy *
 - tifffile *
 - torch >=1.6
 - tqdm *

I'm removing it from pyproject with #248 (I will merge to main as soon as CI passes), which in principle means that any version found on conda will be ok.

As for the QT issue, see:

• Review dependencies after napari-workflows 0.2.7 #225
• Lazy imports for {napari,Qt}-related packages? haesleinhuepf/napari-skimage-regionprops#36

[jluethi]

Great, thanks! :) Can we do the same with the Qt libraries, as we don't depend on them directly and don't want to specify the version required beyond what our dependencies need?

It's not urgent, just thought I'd document my workarounds here and have an issue we can come back on to make this easier :)

[tcompa]

Can we do the same with the Qt libraries, as we don't depend on them directly and don't want to specify the version required beyond what our dependencies need?

We cannot, because of python-poetry/poetry#314.
What we can do is to set an arbitrary version requirement, see 2ead138. With this change, issues with version constraints should be gone.

[jluethi]

Tested this in the current main branch using pip install git+https://github.com/fractal-analytics-platform/fractal-tasks-core.git. pyqt and imagecodecs are still an issue.

I install pyqt manually via conda: conda install pyqt
But then, during fractal-tasks-core installation, it gets stuck at:

Collecting PyQt5
  Using cached PyQt5-5.15.7.tar.gz (3.2 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... \

It's stuck there at 100% CPU load for a while (currently minutes). I'll let it run for a while, let's see if that fixes it. If it doesn't manually removing the PyQt5 dependency from pytoml does allow the installation to run.
Also, napari has a way to be installable after a conda installation of pyqt. Not sure why it's different here and getting stuck at metadata preparation. But I don't think it's worth going to far down this road if we can drop PtQt in the near-ish future as a dependency anyway.

The imagecodecs likely will work, but need to test after it gets past pyqt.

[Those are very clearly issues with PyQt & imagecodecs, not the fractal-core-tasks. Just annoying we have to deal with them]

---

Edit: The PyQt5 installation eventually timed out and got killed. So even the relaxed PyQt requirements and manual conda installation aren't enough yet.

[jluethi]

For the record:
Removing PyQt from the pyproject.toml and installing it separately via conda seems to do the trick for me. imagecodecs is fine now with the relaxed constraints.

To achieve this:
Create an env & activate
git clone the tasks repo
edit the pyproject.toml
conda install pyqt (or e.g. mamba install napari)
cd fractal-tasks-core
pip install .[tools] # this also installs dependencies for tools