Python test failure - installation success unclear from examples

[NiallJeffrey]

I've installed the C package, from which all tests apart from one (TEST 20/55 nmt:he_qdisc [SKIPPED]) passed. However, after the python setup, 22/48 unit tests fail.

It's difficult to tell from the readthedocs examples what the outputs plots are meant to look like (it would be good to plot them after the code snippets). When comparing with full sky anafast outputs, I get a completely different answer with NaMaster.

The install/testing output:
installation_output.txt

My installlation script:
export CFLAGS='-I/usr/local/Cellar/gsl/2.5/include/ -I/usr/local/Cellar/fftw/3.3.8/include/ -I/usr/local/Cellar/cfitsio/3.450/include/ -I/usr/local/Cellar/healpix/3.40/include/ -I/Users/ucapnje/Documents/software/libsharp/auto/include/'
export LDFLAGS='-L/usr/local/Cellar/cfitsio/3.450/lib/ -L/usr/local/Cellar/gsl/2.5/lib/ -L/Users/ucapnje/Documents/software/libsharp/auto/lib/ -L/usr/local/Cellar/healpix/3.40/lib/ -L/usr/local/Cellar/fftw/3.3.8/lib/ -L/Users/ucapnje/Documents/software/NaMaster-master/install_dir/lib/'
./configure --prefix=/Users/ucapnje/Documents/software/NaMaster-master/install_dir
make
make install
make check
export LDFLAGS='-L/usr/local/Cellar/cfitsio/3.450/lib/ -L/usr/local/Cellar/gsl/2.5/lib/ -L/Users/ucapnje/Documents/software/libsharp/auto/lib/ -L/usr/local/Cellar/healpix/3.40/lib/ -L/usr/local/Cellar/fftw/3.3.8/lib/ -L/Users/ucapnje/Documents/software/NaMaster-master/install_dir/lib/'
python setup.py install
python -m unittest discover -v

[beckermr]

This doesn't help yet, but this will be on conda soon!

conda-forge/staged-recipes#7194

[damonge]

@NiallJeffrey sorry for the delay. Have you managed to sort this out yet?
It is very strange that the C tests pass but not the python ones, since most of the latter are almost copies of the former. If the problem is still there, I'd suggest to first try installing it again with the latest version on master.

If the problem persists, let me know and we can run a couple of quick tests to see if things are actually working.

Note that if you're computing power spectra on a cut sky, you would not expect namaster and anafast to return the same answer (you probably know this, but just in case).

[NiallJeffrey]

@damonge I didn't need the code urgently, so I decided to wait for the conda version.

[beckermr]

Ahhhh sorry! The conda version is done!

conda install -c conda-forge namaster

should do the trick!

[beckermr]

This will give you the C version and the python package.

[damonge]

@NiallJeffrey can you confirm if this solved your issue?

[NiallJeffrey]

I am afraid not. I admit not spending a great deal of time trying to solve it, but using the conda version I get "ImportError: No module named '_nmtlib'" and "Reason: image not found".

conda install -c conda-forge namaster
Collecting package metadata: done
Solving environment: done

## Package Plan ##
environment location: /Users/ucapnje/anaconda3
added / updated specs:
- namaster

The following NEW packages will be INSTALLED:

namaster conda-forge/osx-64::namaster-0.9-py36h94a39f0_1001
Proceed ([y]/n)? y

Preparing transaction: done Verifying transaction: done Executing transaction: done
(base) eduroam-int-dhcp-97-155-200:namaster_test ucapnje$ python
Python 3.6.8 |Anaconda, Inc.| (default, Dec 29 2018, 19:04:46)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pymaster
Traceback (most recent call last):
File "/Users/ucapnje/anaconda3/lib/python3.6/site-packages/pymaster/nmtlib.py", line 18, in swig_import_helper
fp, pathname, description = imp.find_module('_nmtlib', [dirname(__file__)])
File "/Users/ucapnje/anaconda3/lib/python3.6/imp.py", line 297, in find_module
raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named '_nmtlib'

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/ucapnje/anaconda3/lib/python3.6/site-packages/pymaster/__init__.py", line 40, in <module>
from pymaster import nmtlib as lib
File "/Users/ucapnje/anaconda3/lib/python3.6/site-packages/pymaster/nmtlib.py", line 28, in <module>
_nmtlib = swig_import_helper()
File "/Users/ucapnje/anaconda3/lib/python3.6/site-packages/pymaster/nmtlib.py", line 20, in swig_import_helper
import _nmtlib
ImportError: dlopen(/Users/ucapnje/anaconda3/lib/python3.6/site-packages/_nmtlib.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libssl.1.0.0.dylib
Referenced from: /Users/ucapnje/anaconda3/lib/libssh2.1.dylib
Reason: image not found

[beckermr]

I think you need to make sure conda-forge is first in your channels as opposed to the anaconda defaults. The error here is not the namaster package. The relevant bit is here

`ImportError: dlopen(/Users/ucapnje/anaconda3/lib/python3.6/site-packages/_nmtlib.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libssl.1.0.0.dylib`
` Referenced from: /Users/ucapnje/anaconda3/lib/libssh2.1.dylib`
` Reason: image not found

which is related to something in your conda env.

[NiallJeffrey]

Ah great. I reordered my channels and updated and now this all works fine. I ran a quick test with cut sky and namaster returns the correct power spectrum.

[damonge]

Great! Thanks a lot for reporting this.

[humnaawan]

@damonge @beckermr Hi, so it appears that I'm running into the same issue noted above:

>>> import pymaster
Traceback (most recent call last):
  File "/Users/humnaawan/anaconda3/lib/python3.6/site-packages/pymaster/nmtlib.py", line 18, in swig_import_helper
    fp, pathname, description = imp.find_module('_nmtlib', [dirname(__file__)])
  File "/Users/humnaawan/anaconda3/lib/python3.6/imp.py", line 297, in find_module
    raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named '_nmtlib'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/humnaawan/anaconda3/lib/python3.6/site-packages/pymaster/__init__.py", line 49, in <module>
    from pymaster import nmtlib as lib
  File "/Users/humnaawan/anaconda3/lib/python3.6/site-packages/pymaster/nmtlib.py", line 28, in <module>
    _nmtlib = swig_import_helper()
  File "/Users/humnaawan/anaconda3/lib/python3.6/site-packages/pymaster/nmtlib.py", line 20, in swig_import_helper
    import _nmtlib
ImportError: dlopen(/Users/humnaawan/anaconda3/lib/python3.6/site-packages/_nmtlib.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libcfitsio.5.dylib
  Referenced from: /Users/humnaawan/anaconda3/lib/libnmt.0.dylib
  Reason: image not found

I tried the solution noted above re conda channels, so that this is what my ~/.condarc looks like

channels:
  - conda-forge
  - defaults

but I am still getting the same error. I have reinstalled the package (using conda install -c conda-forge namaster) but I am still having the same issue.

As a reference, here's the install summary:

(base) humnaawan:~: conda install -c conda-forge namaster
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /Users/humnaawan/anaconda3

  added / updated specs:
    - namaster


The following NEW packages will be INSTALLED:

  cfitsio            conda-forge/osx-64::cfitsio-3.470-h389770f_1
  chealpix           conda-forge/osx-64::chealpix-3.30.0-h01d97ff_1001
  fftw               conda-forge/osx-64::fftw-3.3.8-nompi_h5c49c53_1106
  gsl                conda-forge/osx-64::gsl-2.5-ha2d443c_0
  krb5               conda-forge/osx-64::krb5-1.16.3-hcfa6398_1001
  libcurl            conda-forge/osx-64::libcurl-7.64.1-h16faf7d_0
  libedit            conda-forge/osx-64::libedit-3.1.20170329-hcfe32e1_1001
  libsharp           conda-forge/osx-64::libsharp-1.0.0-h1de35cc_1000
  libssh2            conda-forge/osx-64::libssh2-1.8.2-hcdc9a53_2
  namaster           conda-forge/osx-64::namaster-1.0-py36h1c7146f_1


Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done

I'd really appreciate some help.

[beckermr]

Well cfitsio bumped the version of their library from 5 to 8. What I don't get is why conda is insisting on linking to the actual file and not the symlink that should prevent this error.

@damonge do you understand how namaster is linking here?

[beckermr]

in the mean time you can install one version back of cfitsio and that will fix it

[beckermr]

This PR should fix the issue. The new package will be live in a few hours.

conda-forge/namaster-feedstock#6

[humnaawan]

Okay, I downgraded cfitsio to 3.410 and things work. Thanks so much!

[beckermr]

great! once you update your env again after the rebuild is done (~2 hours or so), you should get a new build of namaster and cfitsio that work together just fine.

[damonge]

Sorry I was offline @beckermr , and thanks a lot for fixing this one. Can I check that this is now fixed?

[stevecunnington]

Hi @damonge @beckermr - it looks like I'm getting a related error on this. Having just updated to macOS Catalina I've had to reinstall anaconda and some other packages. I'm reinstalling namaster and install seems to work:

(base) Steves-MBP:~ stevecunnington$ conda install -c conda-forge namaster
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /Users/stevecunnington/opt/anaconda3

  added / updated specs:
    - namaster


The following NEW packages will be INSTALLED:

  cfitsio            conda-forge/osx-64::cfitsio-3.470-h389770f_2
  chealpix           conda-forge/osx-64::chealpix-3.30.0-ha7c7693_1002
  fftw               conda-forge/osx-64::fftw-3.3.8-nompi_h5c49c53_1106
  libsharp           conda-forge/osx-64::libsharp-1.0.0-h1de35cc_1000
  namaster           conda-forge/osx-64::namaster-1.0-py37he2b8767_0


Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done

But importing pymaster gives a similar error as above

(base) Steves-MBP:~ stevecunnington$ python 
Python 3.7.4 (default, Aug 13 2019, 15:17:50) 
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pymaster
Traceback (most recent call last):
  File "/Users/stevecunnington/opt/anaconda3/lib/python3.7/site-packages/pymaster/nmtlib.py", line 18, in swig_import_helper
    fp, pathname, description = imp.find_module('_nmtlib', [dirname(__file__)])
  File "/Users/stevecunnington/opt/anaconda3/lib/python3.7/imp.py", line 296, in find_module
    raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named '_nmtlib'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/stevecunnington/opt/anaconda3/lib/python3.7/site-packages/pymaster/__init__.py", line 49, in <module>
    from pymaster import nmtlib as lib
  File "/Users/stevecunnington/opt/anaconda3/lib/python3.7/site-packages/pymaster/nmtlib.py", line 28, in <module>
    _nmtlib = swig_import_helper()
  File "/Users/stevecunnington/opt/anaconda3/lib/python3.7/site-packages/pymaster/nmtlib.py", line 20, in swig_import_helper
    import _nmtlib
ImportError: dlopen(/Users/stevecunnington/opt/anaconda3/lib/python3.7/site-packages/_nmtlib.cpython-37m-darwin.so, 2): Library not loaded: @rpath/libcfitsio.5.dylib
  Referenced from: /Users/stevecunnington/opt/anaconda3/lib/libnmt.0.dylib
  Reason: image not found

The channels in my .condarc seem to be as suggested above

channels:
  - conda-forge
  - defaults

Has the issue with cfitsio been solved yet or do I need to downgrade this to make things work?

Any help would be great.

[beckermr]

Unclear. Plus the os upgrade is in the mix. Did you reinstall the Xcode command line tools?

[stevecunnington]

I've just opened Xcode and updated that and this didn't solve the problem

[beckermr]

Awesome thanks for checking! Try downgrading cfitsio and lets see if that helps.

[stevecunnington]

What's the easiest way to do this? And what version should I downgrade to?

[beckermr]

Try something like conda install cfitsio==3.430.

[stevecunnington]

Yep, that worked :) thanks @beckermr !

[beckermr]

Of course! So looking at your install above, you have one of the oldest versions of namaster on conda, conda-forge/osx-64::namaster-1.0-py37he2b8767_0

The latest build should have build number (the single int at the end) of 4. See this page here: https://anaconda.org/conda-forge/namaster/files

I don't get how the solver decided on that.

[beckermr]

You might also try forcing the entire env to upgrade if you want.

[beckermr]

Did you change the labels in your channels or something globally in conda?

[stevecunnington]

No, I've made no changes to my channels. Because of the macOS upgrade, my anaconda should just be the latest default installed one using the graphical install - with no changes made.

[beckermr]

hmmmm ok - well let's keep moving then. I might try and get the old packages marked as broken