Installing wrong version of packages

[bsipocz]

This was first reported in astropy/astropy#5094 (comment), the install process erroneously picked up the wrong numpy version (1.9.0b1) from the astropy-ci-extras conda channel rather than a stable released version (e.g. 1.9.3) from the default conda channel.

There are several issues with this, first we should never pick up a beta release unless asked for it with NUMPY_VERSION=prerelease. Also we should always pick up the latest available version of a given package.

As a side feature request: would be nice to find a way to specify the priority order of conda channels for cases when a package is available in several ones.

[astrofrog]

I feel like astropy-ci-extras shouldn't be providing pre-releases of packages for this reason. When did we start adding these? This broke a number of builds for some other projects of mine too.

If we really want to provide pre-releases, I feel like a different channel would be more appropriate.

A quick workaround is to delete all pre-releases from astropy-ci-extras.

cc @mwcraig

[bsipocz]

I think that happened a very long time ago, hardly any updates were made to that channel. My initial workaround is to remove it from the default channels for now.

[pllim]

I wish I saw this sooner! I had been scratching my head at this failure and finally had to manually pin Numpy to 1.10 to get the tests to pass.

[bsipocz]

@astrofrog - It seems that we've been using ci-extras quite frequently, as for e.g. np 1.8 the latest conda astropy is v0.4.
I'm not sure though that testing affiliated packages for stable astropy with v1.0.3 is any better (e.g. the latest on astropy-ci-helpers for np 1.8).

[mwcraig]

I don't recall why 1.9.0b1 was added to the channel, but I can delete it easily. Actually, don't seem to have that superpower on anaconda.org/astropy-ci-extras.

I'm guessing this is causing problems now because of the conda issue @astrofrog turned up (conda/conda#2675).

@astrofrog or someone else with the appropriate super-powers, could you please add a repo to the astropy org on github called conda-channel-ci-extras? We can then hash out what matrix of versions and packages we want in that channel and get it populated. If there is a desire for any pre-release stuff we can change the label from "main" to something else (pre?) and it won't be picked up just by using the channel.

[larrybradley]

It appears travis-ci is having issues again because numpy 1.9.0b1 is being installed from astropy-ci-extras:
https://travis-ci.org/astropy/photutils/builds/137972366

[astrofrog]

I just removed numpy 1.9.0b1 from astropy-ci-extras, can you restart the build?

[larrybradley]

Restarting now....

[astrofrog]

@mwcraig - done: https://github.com/astropy/conda-channel-ci-extras

[astrofrog]

However, I think we should consider carefully whether we really want to maintain astropy-ci-extras in future. The defaults conda channel already now provides an incomplete matrix of versions, but maybe that's enough - maybe we just need to document better which versions can be used together. For example, Astropy 0.4 can be installed with Numpy 1.8 and 1.9. To some extent, I don't know if we need to worry about combinations not provided by the defaults channel, because it means the versions were not co-eval, so they are not common set-ups.

[astrofrog]

Just for some background, we started astropy-ci-extras when conda was still quite new and there wasn't such a range of versions available.

[larrybradley]

@astrofrog The tests are now failing because numpy 1.5.1(!) is being installed from astropy-ci-extras:

https://travis-ci.org/astropy/photutils/jobs/137972381#L462

[larrybradley]

Mysteriously, that line now says numpy-1.9.0b1 (was I too fast to restart or does travis-ci cache packages?), but I swear it said 1.5.1!

[larrybradley]

It did it again. The travis-ci log originally said numpy 1.5.1:

but now it says numpy 1.9.0b1.

In any case, now more tests are failing from numpy version weirdness:
https://travis-ci.org/astropy/photutils/builds/137972366

[pllim]

@larrybradley , I find that pinning Numpy version explicitly in my package .travis.yml bypasses this problem.

[bsipocz]

The other problem that even the new infrastructure fails with astropy 1.0.1 e.g.: https://travis-ci.org/astropy/astroplan/jobs/137956515#L483

So either we should all,

1. collectively drop old numpy support in affiliated packages (given that we can't test them easily) or
2. build the conda package for the latest astropy releases for all supported numpy versions, or
3. we can use pip install astropy for any numpy that is older than the latest at the time of the latest astropy release.

I think for now I would incline to do the 3rd option, or if we can automate it enough do the 2nd (as it's not sustainable to constantly spend to much person hours on it).

[bsipocz]

@pllim - I'm afraid it's not a solution, as simply there are no recent astropy versions built for older numpies. So it's only pure luck if your package passes the tests.

[astrofrog]

@bsipocz - for 2, what about just providing a list of combinations of versions that works, and people can include those combinations in the CI config? Also, won't you get 2 automatically if you pin the numpy version but don't pin the astropy one?

[mwcraig]

For right now, conda should be be pinned to 4.0.8. It sounds like 4.1.0 (released yesterday) broke a remarkable number of things (see the release notes for 4.1.1, which is not yet available as a conda package, and conda/conda#2700, and this mailing list thread).

The travis builds @larrybradley linked to are using conda 4.1.0.

[astrofrog]

+1 to pinning the conda version

[bsipocz]

@astrofrog - Let's say you need astropy 1.2 for your package (e.g. astroplan), but for numpy, you are accepting 1.10 (or even 1.7 as astropy supports it). There won't be a combination for this in the default conda as they don't build against old versions. Obviously the other way around hardly makes any sense (np 1.11 for astropy 0.4).

[bsipocz]

@mwcraig - Thanks for the links. It's very unfortunate that multiple issues are getting mixed together here causing all these similar problems.

@astrofrog - Are you in the process of doing the pinning or shall I?

[mwcraig]

However, I think we should consider carefully whether we really want to maintain astropy-ci-extras in future.

@astrofrog -- I agree that at a minimum the current channel should be cleared out, and repopulated only when and if we have a clear idea of what it is supposed to do.

My understanding is that its only purpose is to provide numpy/astropy for CI purposes for combinations that continuum doesn't build, and that only to save the time of building numpy (and to a lesser extent astropy) each time. So its purpose ins't to provide conda packages for users, it is just to make it easier to test some of the older python/numpy combinations.

If we don't need to do that because of the recent changes you made to the build matrix then maybe we don't need to do this.

[larrybradley]

conda 4.1.1 is now available as a conda package.

[mwcraig]

@larrybradley -- can you try restarting one of the photutils builds that were failing and let's see what happens?

[larrybradley]

Already running. 😄

[larrybradley]

Done, actually. It still failed:
https://travis-ci.org/astropy/photutils/jobs/137972377

[larrybradley]

The failure appears to be because of the issue with astropy 1.0.1 that @bsipocz mentioned above.

[mwcraig]

How painful would it be to change the travis config to pip install astropy (instead of conda install) for that case?

[astrofrog]

@mwcraig - should be as easy as just putting astropy in PIP_DEPENDENCIES and not specifying ASTROPY_VERSION

[mwcraig]

@astrofrog -- could you please give me write or administrative privileges on:

• The github repo conda-channel-ci-extras
• anaconda.org/astropy-ci-extras

Currently leaning towards populating this with channel with only:

• The astropy/numpy versions that show up in the package template and are not currently in the default channel.

[eteq]

@mwcraig - I changed it so that the "conda build maintainers" (which includes you and @bsipocz) have admin on "conda-channel-ci-extras"

(I don't have control over anaconda.org, though)

[astrofrog]

@mwcraig @eteq - I sent you both the astropy-ci-extras details with PGP encryption

[astrofrog]

@bsipocz - this is fixed too hopefully?

[bsipocz]

I'm closing this as we now have the fallback to pip when the astropy version gets messed up terribly, and dealing with ci-extras is a different issue.