Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

wrong version of ncurses #24

Closed
marxide opened this issue Jul 19, 2018 · 11 comments
Closed

wrong version of ncurses #24

marxide opened this issue Jul 19, 2018 · 11 comments
Assignees
@pkgw

Comments

@marxide
Copy link

marxide commented Jul 19, 2018 •
edited
Loading

A fresh install of casacore (currently 2.4.1) and python-casacore for python2.7 seems to point to the wrong version of ncurses. The casacore dependency is ncurses>=6.1,<6.2.0a0 but when attempting to import most casacore modules (e.g. casacore.tables), it seems to expect libncursesw.5 and I get the following error:

import casacore.tables
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/anaconda3/envs/casacore/lib/python2.7/site-packages/casacore/tables/__init__.py", line 60, in <module>
    from .msutil import *
  File "/anaconda3/envs/casacore/lib/python2.7/site-packages/casacore/tables/msutil.py", line 29, in <module>
    from casacore.tables.table import table, taql, _required_ms_desc
  File "/anaconda3/envs/casacore/lib/python2.7/site-packages/casacore/tables/table.py", line 43, in <module>
    from ._tables import (Table,
ImportError: dlopen(/anaconda3/envs/casacore/lib/python2.7/site-packages/casacore/tables/_tables.so, 2): Library not loaded: @rpath/libncursesw.5.dylib
  Referenced from: /anaconda3/envs/casacore/lib/libcasa_tables.2.dylib
  Reason: image not found

This was in a fresh environment created with conda create -n casacore -c conda-forge python=2.7 casacore python-casacore.


Environment (conda list): 
$ conda list
# packages in environment at /anaconda3/envs/casacore:
#
# Name                    Version                   Build  Channel
boost                     1.67.0           py27h3e44d54_0    conda-forge
boost-cpp                 1.67.0               h3a22d5f_0    conda-forge
bzip2                     1.0.6                         1    conda-forge
ca-certificates           2018.4.16                     0    conda-forge
casacore                  2.4.1            py27he94f398_3    conda-forge
certifi                   2018.4.16                py27_0    conda-forge
cfitsio                   3.430                hd130d23_0    conda-forge
curl                      7.60.0               h93b3f91_0    conda-forge
fftw                      3.3.8                h470a237_0    conda-forge
hdf5                      1.10.2               hc401514_1    conda-forge
icu                       58.2                 hfc679d8_0    conda-forge
intel-openmp              2018.0.3                      0
krb5                      1.14.6                        0    conda-forge
libgfortran               3.0.1                h93005f0_2
libgfortran-ng            3.0.1                h93005f0_2
libpng                    1.6.35               ha92aebf_0    conda-forge
libssh2                   1.8.0                h5b517e9_2    conda-forge
libxcb                    1.13                 h470a237_1    conda-forge
mkl                       2018.0.3                      1
mkl_fft                   1.0.2                    py27_0    conda-forge
mkl_random                1.0.1                    py27_0    conda-forge
ncurses                   6.1                           0    conda-forge
numpy                     1.14.2           py27ha9ae307_1
openblas                  0.2.20                        8    conda-forge
openssl                   1.0.2o                        0    conda-forge
pgplot                    5.2.2                hd376476_3    conda-forge
pip                       9.0.3                    py27_0    conda-forge
pthread-stubs             0.4                  h470a237_1    conda-forge
python                    2.7.15                        0    conda-forge
python-casacore           2.2.1            py27h5c5fb89_1    conda-forge
readline                  7.0                  hc1231fa_4
setuptools                40.0.0                   py27_0    conda-forge
sqlite                    3.20.1                        0    conda-forge
tk                        8.6.7                         0    conda-forge
wcslib                    5.18                          2    conda-forge
wheel                     0.31.1                   py27_0    conda-forge
xorg-kbproto              1.0.7                h470a237_2    conda-forge
xorg-libx11               1.6.5                h470a237_1    conda-forge
xorg-libxau               1.0.8                h470a237_5    conda-forge
xorg-libxdmcp             1.1.2                h470a237_6    conda-forge
xorg-xproto               7.0.31               h470a237_7    conda-forge
xz                        5.2.3                         0    conda-forge
zlib                      1.2.11               h470a237_3    conda-forge

Details about conda and system ( conda info ): 
$ conda info

     active environment : casacore
    active env location : /anaconda3/envs/casacore
            shell level : 2
       user config file : /Users/marxide/.condarc
 populated config files : /Users/marxide/.condarc
          conda version : 4.4.10
    conda-build version : 3.4.1
         python version : 3.6.4.final.0
       base environment : /anaconda3  (writable)
           channel URLs : https://repo.continuum.io/pkgs/main/osx-64
                          https://repo.continuum.io/pkgs/main/noarch
                          https://repo.continuum.io/pkgs/free/osx-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/r/osx-64
                          https://repo.continuum.io/pkgs/r/noarch
                          https://repo.continuum.io/pkgs/pro/osx-64
                          https://repo.continuum.io/pkgs/pro/noarch
          package cache : /anaconda3/pkgs
                          /Users/marxide/.conda/pkgs
       envs directories : /anaconda3/envs
                          /Users/marxide/.conda/envs
               platform : osx-64
             user-agent : conda/4.4.10 requests/2.18.4 CPython/3.6.4 Darwin/16.7.0 OSX/10.12.6
                UID:GID : 26778:286781
             netrc file : None
           offline mode : False
@pkgw pkgw self-assigned this Jul 19, 2018
@pkgw
Copy link
Contributor

pkgw commented Jul 19, 2018 •
edited
Loading

Yikes!

It looks like there is a problem because in conda-build 3, the "build environment" installs ncurses version 5, but the "host environment" installs version 6. During the configuration process, cmake decides to link to the host environment's libreadline but somehow decides to pull the dependency on libncursesw from the build environment, getting the wrong version.

This all worked on my machine (and possibly others) because my Linux OS still provides version 5 of libncurses at a system level, so the linking worked out even though the libraries were linking to a non-core library not available in the conda environment.

If I can get cmake to not search the "build environment" for shared library paths, that should fix things. Time to do some digging ...

@pkgw
Copy link
Contributor

pkgw commented Jul 19, 2018

@conda-forge/staged-recipes Tagging since this problem could pop up and cause subtle problems for other cmake-using packages, I think.

@ocefpaf
Copy link
Member

ocefpaf commented Jul 19, 2018

Ping @conda-forge/core

@scopatz
Copy link
Member

scopatz commented Jul 19, 2018

You should be able to rerender the recipe to get the latest ncurses pinning.

@pkgw
Copy link
Contributor

pkgw commented Jul 19, 2018

@scopatz I think the proximate problem here is that the recipe's pinning was newer than that used by some of the build-environment packages; although my local testing is suggesting that the build-env packages have caught up by now. Regardless, I'm pretty sure that nothing about this issue is specific to this particular package beyond its use of cmake, so I worry that comparable hard-to-detect issues will occur in the future.

@CJ-Wright
Copy link
Member

Does this mean we need to be faster getting packages onto the new pinnings?

@pkgw
Copy link
Contributor

pkgw commented Jul 19, 2018

@CJ-Wright That would be nice and would help, I think, but there's always going to be some lag time given how the project operates.

I think I have identified a pretty straightforward way to avoid the problem going forward, though. Testing is ongoing.

@CJ-Wright
Copy link
Member

Ah ok. The bot has some pin-shift-rebuild capabilities but they haven't been put into production yet. I'm trying to gauge need and such so I can do some prioritization.

Attn:
@justcalamari

pkgw added a commit to pkgw/casacore-feedstock that referenced this issue Jul 19, 2018
@pkgw
Hack CMake to ignore the "build environment" prefix in its searches
fd9ab23 
In [conda-forge GitHub issue conda-forge#24](conda-forge#24)
there was a problem because CMake decided to link with a version of the
ncurses library found in the "build environment" prefix, rather than the "host
environment" -- the major versions were different and so the resulting
binaries did not link against the major version intended in the recipe
specification.

As far as I can tell, the problem is that CMake decides that the "build
environment" library directory should be searched for shared libraries, which
is in turn because the build-env prefix shows up in the variable
`CMAKE_SYSTEM_PREFIX_PATH`. We hack the CMakeLists.txt file to remove the
relevant directory, which fixes the problem. As a bonus, a bunch of messages
about libraries like `libz` being shadoweded between the build and host
prefixes now go away.

The [CMake
docs](https://cmake.org/cmake/help/latest/variable/CMAKE_SYSTEM_PREFIX_PATH.html)
say that you're not supposed to futz with this variable, but I couldn't find
an alternative solution. It may be fixable by patching our cmake package --
although I don't have a good sense as to whether we might sometimes want the
build-env prefix to be in `CMAKE_SYSTEM_PREFIX_PATH`.
@pkgw pkgw mentioned this issue Jul 19, 2018
Merged
4 tasks
@pkgw
Copy link
Contributor

pkgw commented Jul 19, 2018

First-cut workaround PR filed.

@pkgw
Copy link
Contributor

pkgw commented Jul 19, 2018

OK @marxide, a build should be coming along that fixes this.

BTW, in the CI output for the build behind the most recent deployed version, conda inspect indeed pointed out the libncurses.5.dylib was "not found" ... it was always a bit weird to me that this was not a build-killing error. But I guess we're moving away from conda inspect anyway, right?

@pkgw pkgw closed this as completed Jul 19, 2018
@marxide
Copy link
Author

marxide commented Jul 20, 2018

Many thanks! Latest build seems to fix this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pkgw pkgw

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@pkgw @scopatz @ocefpaf @CJ-Wright @marxide