Required system libraries for packages

[bekozi]

What is the best practice for indicating required system libraries for a conda package?

For instance, when installing ESMF on Ubuntu, build-essential and gfortran are required (i.e. apt-get install build-essential gfortran). Also, libexpat1 is required for udunits2. I am thinking expat should maybe be a requirement for udunits2?

Thanks!

[ocefpaf]

That is a good question! We need to make that clearer somehow...

Right now there is no clear way to declare them for all CIs. We add them separately in the .travis.yml for OS X and the Dockerfile for Linux. On Windows we must package them! (See the expat and the udunits2 recipes for an example.)

[bekozi]

I see! It would be nice to have platform pages where recommended packages could be listed for yum / apt-get / brew, etc. Package-specific might be too much to manage, but a list of recommended Ubuntu packages (for example) seems reasonable. Could sections be added in the "Setting up the IOOS Python Environment for..." pages?

[ocefpaf]

package-specific might be too much to manage

It is indeed but necessary for build reproducibility.

Could sections be added in the "Setting up the IOOS Python Environment for..."

We can certainly improve the docs, but ideally no system library is needed to use the packages (only to build them). I am trying to eliminate that problem when it crops up. (See #667 for example.)

[bekozi]

We can certainly improve the docs, but ideally no system library is needed to use the packages (only to build them).

Ahhh, okay. Thanks for the clarification. So for ESMF this installation test fails:

docker run continuumio/miniconda bash -c 'conda install -c ioos esmpy && python -c "import ESMF; m = ESMF.Manager(); print(m.pet_count)"'

Traceback (most recent call last):
  File "/opt/conda/lib/python2.7/site-packages/ESMF/interface/loadESMF.py", line 111, in <module>
    mode=ct.RTLD_GLOBAL)
  File "/opt/conda/lib/python2.7/ctypes/__init__.py", line 365, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libgfortran.so.3: cannot open shared object file: No such file or directory
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/lib/python2.7/site-packages/ESMF/__init__.py", line 66, in <module>
    from api.esmpymanager import *
  File "/opt/conda/lib/python2.7/site-packages/ESMF/api/esmpymanager.py", line 10, in <module>
    from ESMF.interface.cbindings import *
  File "/opt/conda/lib/python2.7/site-packages/ESMF/interface/cbindings.py", line 12, in <module>
    from ESMF.interface.loadESMF import _ESMF
  File "/opt/conda/lib/python2.7/site-packages/ESMF/interface/loadESMF.py", line 114, in <module>
    raise ValueError('The ESMF shared library did not load properly.')
ValueError: The ESMF shared library did not load properly.

This will pass because we are installing gfortran (turns out for this test case build-essential is not needed:

docker run continuumio/miniconda bash -c 'apt-get -y update && apt-get -y install gfortran && conda install -c ioos esmpy && python -c "import ESMF; m = ESMF.Manager(); print(m.pet_count)"'

From what you said, we would want to solve this upstream in the recipe - trying to build with conda's libgfortran for example. Is this correct? In the past, we addressed this by providing our own "gcc-with-gfortran" package. This was before conda provided a Fortran compiler.

[ocefpaf]

From what you said, we would want to solve this upstream in the recipe - trying to build with conda's libgfortran for example. Is this correct?

Yes, if possible, if is does not work...

In the past, we addressed this by providing our own "gcc-with-gfortran" package. This was before conda provided a Fortran compiler.

.. we will go with what you know to work 😉

The goal is to use as much as we can from the default channel. That way we avoid putting an extra burden on the channel maintainer.

Ideally, if we need something that is"broken" in the default channel we should push a fix to the default channel rather than maintaining our own build.

There is a significant increase in channels out there, providing different recipes for the same package. I think that this is harmful to the scientific community and we should strive to avoid that kind of "recipe forking" as much as we can by pushing changes to community channels and/or the default channel.

[ocefpaf]

@bekozi can you test if installing libgfortran fixes that? If so we can just add is as a dependency rather than re-building it with conda's or a custom gcc.

[bekozi]

Installing libgfortran doesn't fix the issue (tried that already). There will need to be some rebuilding and/or re-linking. Let me do some experimentation with the default channel compilers and see how it goes... Wish me luck. :-)

[ocefpaf]

Installing libgfortran doesn't fix the issue (tried that already).

😞

Let me do some experimentation with the default channel compilers and see how it goes... Wish me luck. :-)

Finger crossed!

[bekozi]

I've done some experimenting with Linux build set-ups. OSX and Windows may have their own set of challenges.

1. The default channel gcc now provides gfortran. Nice. At the very least, the default channel gcc requires libc6-dev on Ubuntu and glibc-devel on CentOS6.
2. If a base gcc/gfortran installation is not expected on user machines, netcdf-fortran and ESMF need to be rebuilt with gcc as a build/run dependency.
3. The default channel mpich2 has Fortran compiler linking issues when attempting to build ESMF (probably anything else as well). Building a new mpich2 using the default channel gcc addresses the issue. mpi4py may also need to be built.

The draft changes/additions to build recipes can be viewed here: master...NESII:esmf-libgfortran.

There are two paths to addressing this issue from these tests. To be clear, the issue is users must have gfortran installed on their machine before using IOOS's ESMF.

1. Provide a location in the wiki for platform-specific documentation on how to install gfortran.
2. Add default gcc requirement to at least netcdf-fortran and ESMF. Either build mpich2 with default gcc or fix the default channel mpich2.

Option 1 is definitely easier but not as inclusive. Option 2 seems more durable in the long-run. It may be painful to implement. There may be cascading issues in other packages by introducing gcc. Option ? could involve fancy post linking.

What do you think?

[ocefpaf]

Great work @bekozi! It is sad that the default channel "build tools" are not ideal yet.

I am 👍 for option (1)

[bekozi]

Me too! I'd recommend text for Linux like below. I'd send a PR, but I don't think that's possible for wikis... Happy Holidays, BTW! 🎄

Prerequisite System Libraries

Most IOOS conda packages have no prerequisite system libraries. Exceptions are listed in the table below.

Prerequisite	Installation Instructions	Required By
GFortran	Install GFortran	esmf, esmpy

Install GFortran

Linux

GFortran may be installed using your default Linux distribution's package manager:

$ [sudo] apt-get install gfortran
or
$ [sudo] yum install gcc-gfortran

MacOSX

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Phasellus ac erat sem. Integer volutpat nibh ipsum. Nam posuere tempor sem eu venenatis. Aenean vitae semper odio, id posuere diam. Quisque in nunc interdum turpis accumsan pellentesque. Nullam ac diam nec nulla vulputate semper.

Windows

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed consectetur elit tellus, sed laoreet justo vulputate rhoncus. Fusce laoreet non massa at aliquet. Curabitur efficitur lacus vitae ex iaculis, nec luctus eros lobortis. Fusce bibendum lorem orci, vitae rutrum leo commodo consectetur.

[ocefpaf]

Sounds great! Thanks Ben.

I will leave that to our wiki master @rsignell-usgs 😜

Happy Holidays!

[bekozi]

@rsignell-usgs: Have you had a chance to look over these suggested wiki additions? #676 (comment) Thanks!

[ocefpaf]

@bekozi I added a few extra dependencies to the new packages on conda-forge and this problem should be fixed. If not let's open a new issue there.