install failing due to links

[david-murr]

Install is failing at:

IOError: [Errno 2] No such file or directory: '/Users/xxx/anaconda/pkgs/hdf5-1.8.17-1/bin/activate'

A quick look shows that these have been installed as links (to nonexistent directories)
lrwxr-xr-x  1 xxxx  staff      37 Jun 21 19:03 activate -> /Users/travis/miniconda3/bin/activate
lrwxr-xr-x  1 xxxx  staff      34 Jun 21 19:03 conda -> /Users/travis/miniconda3/bin/conda
lrwxr-xr-x  1 xxxx  staff      39 Jun 21 19:03 deactivate -> /Users/travis/miniconda3/bin/deactivate

condo info
Using Anaconda Cloud api site https://api.anaconda.org
Current conda install:

             platform : osx-64
        conda version : 4.1.2
  conda-build version : 0+unknown
       python version : 2.7.11.final.0
     requests version : 2.9.1
     root environment : /Users/xxxx/anaconda  (writable)
  default environment : /Users/xxxx/anaconda
     envs directories : /Users/xxxx/anaconda/envs
        package cache : /Users/xxxx/anaconda/pkgs
         channel URLs : https://conda.anaconda.org/conda-forge/osx-64/
                        https://conda.anaconda.org/conda-forge/noarch/
                        https://repo.continuum.io/pkgs/free/osx-64/
                        https://repo.continuum.io/pkgs/free/noarch/
                        https://repo.continuum.io/pkgs/pro/osx-64/
                        https://repo.continuum.io/pkgs/pro/noarch/
          config file : /Users/xxxx/.condarc
    is foreign system : False

[gillins]

This is quite serious as conda gets overwritten breaking the user's environment. I can confirm same thing on my Mac.

Is this a problem with the toolchain package @jakirkham ? There it seems to some copying of scripts in its build.sh.

[jakirkham]

Hmm... I don't think so. The toolchain has UNIX scripts are named toolchain_activate.sh not activate. Also, conda-build normally avoids files that came from a dependency unless they are explicitly exempted (e.g. libgcc). As we didn't add these files or exempt them, I don't think it was an error in our packaging.

Though there were some changes to conda,conda-env andconda-build recently to handle environments differently. This process adds symlinks to the activate, deactivate, and root, which are added to the bin directory of each environment. Unfortunately, it may be bundling them up when conda-build packages everything. If that's the case, this would not be limited to hdf5 though. I would expect that the most recent version of conda-build does not have this problem. This can be confirmed by looking at Linux packages from build number 1, which don't have this problem and do use the latest conda-build.

However, we are not using the most recent version on Travis CI. We could try updating to the most recent version and see if that makes a difference. However, if my suspicions are correct, this affects all OS X packages that we have deployed recently. A cursory glance at some other packages confirm this is the case.

[jakirkham]

@ocefpaf, it looks like we have a serious issue on our hands. We need to unpin conda-build on Travis CI.

[jakirkham]

Sorry for the inconvenience @david-murr. Please see this PR ( #25 ). I suspect this will fix the issue.

[jakirkham]

@msarahan, does my reasoning above sound correct to you?

[msarahan]

SGTM. Very strange that this is happening. Here's the deal to keep in mind, though. There are two "generations" to be wary of:

Conda <=4.0.x & conda-env <=2.4.5
Conda >=4.1.x & conda-env >=2.5.0

The former has activation scripts in conda-env. The latter has activation scripts in conda. Upgrades and downgrades are not always 100% as smooth as we would like.

As for packaging the activation scripts incorrectly, that might actually be a new bug with activating environments during build. I need to run some tests and make sure...

[jakirkham]

As for packaging the activation scripts incorrectly, that might actually be a new bug with activating environments during build. I need to run some tests and make sure...

I don't think it is because the hdf5 builds from the same generation on CircleCI (no conda-build pinning) don't have this issue.

[msarahan]

I'm hopeful that you're right, but I'll write a test anyway. It can't hurt to prevent that from happening if it isn't already.

[jakirkham]

Sure, feel free to test. Though I think this is working correctly from the examples I have seen.

[jakirkham]

When you do test this, @msarahan, I would recommend having some cases that don't install things in bin and some that do. It seems that might catch some corner cases.

[jakirkham]

To provide another example, take a look at jsoncpp-0.10.6-0.tar.bz2 for osx-64. It should have nothing in bin, but has also included the symlinks for activate, conda, and deactivate.

[jakirkham]

Adding you on this one @pelson so that you are aware of this issue. Basically our working theory is the conda-build pinning on Travis CI is resulting in broken packages.

[jakirkham]

Until we get this sorted, @david-murr, please do conda install hdf5=1.8.17=0.

[jakirkham]

We have made a fresh build of hdf5 and this build does not have the troublesome symlinks.

cc @dopplershift

[jakirkham]

Was able to download and save a simple HDF5 file with it.

[gillins]

geos seems good also. Thanks for sorting this out @jakirkham

Should we work through the other packages next? Or wait for conda-forge/toolchain-feedstock#9 ?

[jakirkham]

geos seems good also. Thanks for sorting this out @jakirkham

Glad to help, @gillins. 😄 Glad to hear it looks good.

Should we work through the other packages next?

Sure. We should look at other ones. Though I can't promise too much more effort tonight before needing to call it. 😉

Was looking back at this issue ( conda-forge/numpy-feedstock#15 ) to get some idea of what needs to go next. Would it be libkea and then gdal? Are there others we need to be aware of?

The nice thing about all the great work that @msarahan has been doing on environment activation is this process should get easier for us.

Or wait for conda-forge/toolchain-feedstock#9 ?

I'd really like to hear something from someone at Continuum on whether this matters to them or if we can make it easier for them to reuse our source. That being said, it has gotten a lot of positive response. So, I do think we will go through with it one way or another. Though I don't really want to leave stuff in a broken state while that gets settled. Let's see what we can do with the remaining libraries.

[gillins]

Yes libkea and gdal should be next, I should be able to do them a bit later today. I've also noticed xercesc will need the same treatment. I'm really only focused on gdal related packages, so there are bound to be others.

Is it just C++ projects we need to add the toolchain to? Or C ones also?

[jakirkham]

Really anything that is compiled should use the toolchain. Though the issues on Mac arise w.r.t. C++ (mixing libc++ and libstdc++). The toolchain should force libc++. FWICT C runtimes don't have this problem so mixing clang and gcc binaries should be ok. That being said, we may want to move those to clang too just to avoid confusion.