Move All Package Dependencies to a Single Place (Trac #681)

[ajj]

check what the right dependencies are for each package (sasmodels, sasview) and make sure setup.py calls the required ones.

Can we make it have optional requirements? e.g. is matplotlib actually required for sasview to function (might not want lots of dependencies for deployment on cluster/aws etc)

Migrated from http://trac.sasview.org/ticket/681

{
    "status": "assigned",
    "changetime": "2019-03-24T15:31:39",
    "_ts": "2019-03-24 15:31:39.490479+00:00",
    "description": "check what the right dependencies are for each package (sasmodels, sasview) and make sure setup.py calls the required ones.\n\nCan we make it have optional requirements? e.g. is matplotlib actually required for sasview to function (might not want lots of dependencies for deployment on cluster/aws etc)",
    "reporter": "ajj",
    "cc": "",
    "resolution": "",
    "workpackage": "SasView Bug Fixing",
    "time": "2016-10-04T11:36:12",
    "component": "SasView",
    "summary": "Move All Package Dependencies to a Single Place",
    "priority": "critical",
    "keywords": "",
    "milestone": "Admin Tasks",
    "owner": "piotr",
    "type": "task"
}

[ricleal]

Trac update at 2016/10/09 16:37:33: ricardo commented:

This may help...
Below is the pip freeze of my virtual environments:

'''SasView'''

bumps==0.7.6
cycler==0.10.0
h5py==2.6.0
lxml==3.6.4
matplotlib==1.5.3
numpy==1.11.2
periodictable==1.4.1
Pillow==3.4.1
pyparsing==2.1.10
python-dateutil==2.5.3
pytz==2016.7
scipy==0.18.1
six==1.10.0
wxPython==3.0.2.0
wxPython-common==3.0.2.0

'''SasModels'''

appdirs==1.4.0
cffi==1.8.3
cycler==0.10.0
decorator==4.0.10
matplotlib==1.5.3
numpy==1.11.2
py==1.4.31
pycparser==2.14
pyopencl==2016.1
pyparsing==2.1.10
pytest==3.0.3
python-dateutil==2.5.3
pytools==2016.2.2
pytz==2016.7
scipy==0.18.1
six==1.10.0

Matplotlib is for compare.sh only I guess...

[pkienzle]

Trac update at 2016/10/10 16:50:22: pkienzle commented:

Is sasmodels mature enough to release as a separate pypi component? Then it becomes just another dependency as far as sasview is concerned, with the same coordination requirements as bumps. May want to move sasview_model.py from sasmodels to sasview first, but not required.

[butlerpd]

Trac update at 2016/11/08 14:32:56:

• butler changed description from:

check what the right dependencies are for each package (sasmodels, sasview) and make sure setup.py calls the required ones.

Can we make it have optional requirements? e.g. is matplotlibe actually required for sasview to function (might not want lots of dependencies for deployment on cluster/aws etc)

to:

check what the right dependencies are for each package (sasmodels, sasview) and make sure setup.py calls the required ones.

Can we make it have optional requirements? e.g. is matplotlib actually required for sasview to function (might not want lots of dependencies for deployment on cluster/aws etc)

• butler changed milestone from "SasView 4.1.0" to "Admin Tasks"
• butler changed type from "defect" to "task"

[rozyczko]

Trac update at 2016/11/17 15:10:20: piotr commented:

I started working on this and soon it became apparent that the scope is much wider, especially considering move to Anaconda.

We want a set of package definitions (name-version) which developers can modify and which will be used on the build server to create appropriate environment in which sasview and sasmodels are built.

This means combining conda channels, pip installation and local packages from Gohlke's archive.

[butlerpd]

Trac update at 2016/11/20 02:09:35:

• butler changed owner from "" to "piotr"
• butler changed status from "new" to "assigned"

[pkienzle]

Trac update at 2017/03/16 16:05:55: pkienzle commented:

Basically, anything which depends on significant third party libraries might be a problem if pip tries to install it. Pypi has binary wheels for some of these, with more available for windows than for mac. Anaconda has most of them, as do other python environments for windows.

They are:

pillow: libjpeg, libz, libtiff, libfreetype, tcl, tk
numpy: blas/lapack, fft, umfpack, ...
scipy: ...
wx: libwx, ...
lxml: libxml2, libxslt
h5py: hdf5
reportlab: freetype
pyopencl: libOpenCL

The remaining direct requirements have no C dependencies and can be installed via pip. Some of them may also be available as system packages in linux or anaconda:

pyparsing
html5lib
pylint
xhtml2pdf (depends on pillow and reportlab)
sphinx
unittest-xml-reporting

We are supporting bumps and periodictable; they can be pip installed or git cloned, depending on whether you want to work on them:

periodictable
bumps

As Piotr's list above show, there are some indirect dependencies as well:

appdirs
cycler
decorator
py
pycparser
pytest
python-dateutil
pytools
pytz
six

OpenCL is a bit special since it is not available in anaconda or as binary wheels. I think "pip install pyopencl" worked for the mac (it already has the required drivers, headers and libraries). For windows I had to install the drivers and then used prebuilt binaries.* On linux I built my own packages. Ubuntu has packages for the headers and libraries, but I don't know about Fedora. Vendor drivers can be a bit challenging on linux...

• Windows pyopencl binaries: http://www.lfd.uci.edu/~gohlke/pythonlibs/#pyopencl

[butlerpd]

Trac update at 2019/03/24 15:31:39:

• butler commented:

Much of this has been superseded with Anaconda and yml files. However we now have 5 files each "hard coding" the dependencies.

• apple yml
• unix yml
• windows yml
• installers/Check_packages.py
• build_tools/requirements.txt

They do have different roles but we should just have maybe the requirements file and import that information into each of the other places.

• butler changed priority from "major" to "critical"
• butler changed summary from "Package dependencies" to "Move All Package Dependencies to a Single Place"

[rozyczko]

This is something we've been putting a lot of effort locally. in the 5.0 branch we use platform-specific (although very similar) yaml files for sourcing packages. This is done with explicit versioning, so potential for conflicts is minimized.
We do upgrade versions of major modules (numpy, mpl, pyinstaller etc) but do it as a specific release task rather than on continuous basis.
As for pyopencl, it is available in conda-forge as of 2018, so we just pull it from there.