Everything is broken: upgrade conda to version 4.4+

[dhimmel]

I had anaconda3 installed with the default channels.

Then I added conda-forge to my config as suggested in https://conda-forge.org/:

conda config --add channels conda-forge 

I did this because I like conda-forge and would like my packages to come from conda-forge rather than the less open anaconda channel.

However, this downgraded conda:

    conda:                              4.5.0-py36_0                           --> 4.3.34-py36_0                 conda-forge

Now there's a huge mess. The following line in my .bashrc no longer works:

source ~/anaconda3/etc/profile.d/conda.sh

conda activate no longer works. Most conda commands now throw errors. I believe this is due to how conda activation changed in 4.4. Given this upgrade shouldn't it be a high priority to update conda in conda-forge!

[scopatz]

Check out #41 and #42

[dhimmel]

Check out #41 and #42

Yeah I saw those! I think either would close this issue.

Just wanted to post it separately because I expect others will run into this problem and didn't want to sabotage focused PR discussion.

[scopatz]

Ok thanks!

[jakirkham]

Yeah the challenge is updating related infrastructure in conda-forge. I'd like the larger conda community to get to a point where updating conda related tools anywhere is no big deal as that will be a much healther place to be (including for conda-forge users and within conda-forge infrastructure). Think this really depends on having stable (basically unchanging) API from these tools that we use exclusively to move this forward. There have been improvements on this front for sure, but there is definitely still room for growth. This came up in the core meeting the other day.

cc @conda-forge/core @kalefranz @msarahan

[mingwandroid]

Think this really depends on having stable (basically unchanging) API from these tools that we use exclusively to move this forward.

The elephant in the room is that conda-forge could consider not providing packages for these things at all.

[jakirkham]

I fail to see how that solves the problem. An updated conda from any channel currently will break the conda-forge infrastructure. Removing the conda package from conda-forge would only remove the control lever that is holding things together currently.

[djsutherland]

So conda-smithy and whatnot is pinned to appropriate versions of conda / conda-build / etc, right? Is it possible to just tweak the build scripts so that they get an appropriate version of the things they need, but regular users can still have current latest versions from conda-forge? (Don't really know anything about the way these things break, so maybe that's hard / not possible for some reason.)

[mingwandroid]

Right, so if conda only existed in defaults we'd have to update things in lock-step with conda-forge, that goes without saying though.

[jakirkham]

So conda-smithy and whatnot is pinned to appropriate versions of conda / conda-build / etc, right? Is it possible to just tweak the build scripts so that they get an appropriate version of the things they need, but regular users can still have current latest versions from conda-forge?

Unfortunately that doesn't work either.

conda will weasel out of version constraints (particularly ones applied to conda). For example it might find an older version of conda-smithy that doesn't have the pin for conda and install it. We could mark all of these as broken (reasonable complaints about this strategy aside), but we will end up with more problems downgrading conda as reported by the user above. We have tried to improve the situation by having conda-smithy check the conda version and error out when it is incompatible, but it is IMHO still subpar.

We could look into pinning things in conda-forge-build-setup. It will still cause issues for users using conda-smithy locally. Not sure how we would want to address that.

[jakirkham]

As always Ray, I'm in favor of closer collaboration with defaults.

Am not sure that the answer is just delete conda from conda-forge though as there still remain two very important issues. Regardless of where conda comes from.

1. Outside tooling needs a stable API.
2. conda needs to respect version constraints of packages more.

[mingwandroid]

Yeah, I wasn't suggesting it completely seriously, but wanted to float the idea again as a possible distant goal we could maybe one day consider. Sorry for the off-topic-ish-ness!

[djsutherland]

So finding old versions that don't have a version constraint is really a bug in the old packages, and marking them as broken seems reasonable to me. It's not like lots of people should be needing to install the old versions now anyway...

Of course stable API would solve all of these problems in any case, but given the world we live in right now, it seems to me like marking the old versions as broken and changing conda-forge-build-setup to install conda-smithy at the same time as updating conda and conda-build would allow us to release current versions of conda for users, right?

And yes, there would still be issues with local users of conda-smithy, but now that rerendering can be done by the webservice I think that's probably a relatively small group of people?

[jakirkham]

Ah ok. Thanks for clarifying Ray.

I'd hope that the syncing up core Anaconda recipes would help that problem and as noted there have been improvements on the API issue on both ends. It just remains incomplete ATM. There are some finer points that we might want to discuss out-of-band.

[kalefranz]

I also see this issue as one of pinning the build tools appropriately. It seems to me suboptimal to couple the entire community of users who typically have no interest in the particular build tools to only supported versions of build tool dependencies.

Once we’re able to get conda-forge on conda 4.4 (actually we released 4.5 yesterday), conda-smithy and other build tools could probably make good use of run_constrained and constrained dependencies.

[jakirkham]

If you would like to give a PR to conda-forge-build-setup a try @dougalsutherland, that certainly would be a welcome start and useful regardless the outcome of the rest of the discussion.

[jakirkham]

Right so this biggest hurtle has been conda likes to select old build number packages if it will get newer versions. This causes many of the pains discussed above and mentioned in the past. Have there been improvements on this front?

Tracking selected vs. merely pulled in dependencies, as I understand conda 4.4 to do, will help the problem as well. Though I don't know how a conflict between conda and conda-smithy in terms of version constraints will shake out here. Would be interesting to hear thoughts on this or maybe even see an experiment.

@msarahan, @isuruf, and I spent a fair bit of time getting conda-smithy compatible with conda-build 3 without directly using conda or conda-build-all, which should reduce the amount of code it is touching in other tools and hopefully make it more resilient in the future. The other issue along these lines is getting staged-recipes to just use conda-build w/o conda-build-all. That is still WIP last I checked.

[dhimmel]

Solution

For others who are having this issue, I resolved it by switching the order of the channels in ~/.condarc to become:

channels:
  - defaults
  - conda-forge

Then I ran

~/anaconda3/bin/conda update --all

which made the following change to conda:

    conda:                              4.3.34-py36_0                 conda-forge --> 4.5.0-py36_0              

Question

I'm not an expert on conda, so most of the discussion in this issue is over my head. However, one thing that would have prevented my issue is if I could install miniconda/anaconda directly from conda-forge. As conda-forge grows, I think a lot of users (myself included) will want to interface only with the community driven & open source conda-forge rather than the partially-proprietary continuum anaconda. Would it make sense to have a conda-forge installer where the default channel is conda-forge? In other words, can you install miniconda and have that contain the conda-forge version of conda right off the bat?

[mingwandroid]

Please consider carefully what the likely outcome is if the community forks in such a fashion, keeping in mind that the recipes used to build the packages for AD are forked from conda-forge and that Anaconda Inc. pays the salaries of the developers working on the core tools behind conda-forge.

partially-proprietary continuum anaconda

Please explain what is proprietary about the Anaconda Distribution. I could argue that since conda-forge packages are built on 3 different proprietary CI systems we are less proprietary.

The relationship should be symbiotic and we need to work closer together not further apart.

Our team have loads of great PRs we'd love to be able to push to conda-forge and we hope to make more of our infrastructure available to conda-forge as time progresses. We are really not interested in forging our own path here (pun intended).

[kalefranz]

partially-proprietary continuum anaconda

We really don’t know what you mean here. To my knowledge, there’s nothing I can think of that’s proprietary with what Anaconda distributes through repo.continuum.io. Every package there has the full conda recipe embedded in it. And the recipes are also all at github.com/AnacondaRecipes. That github org is set up just like conda-forge, so that we can easily push all work up to conda-forge. In fact most everything there was forked from conda-forge, and we’ve been building with new compiler technology and conda-build 3 on top. The compiler technology and conda-build 3 aren’t being kept from conda-forge at all. We’re putting in our own effort to get conda-forge’s build pipelines compatible, so we can all be on the same page. To the extent we’re distribute proprietary software in repo.continuum.io, its not proprietary Anaconda software. It’s purely for the community’s benefit. MKL is the best example. We’ve put in effort to secure a special licensing arrangement from Intel so that we can distribute out to the world at no charge to users their proprietary blas implementation that dramatically improves computation times for people using numpy. So yeah I guess there’s some proprietary stuff, if you take that perspective, but then we also distribute openblas if you really want it.

My rant here isn’t to throw shade on conda-forge at all. We love conda-forge. And we contribute a lot if resources and time. I just want to be sure that we’re not allowing inaccuracies to propogate regarding the Anaconda repositories somehow being proprietary.

[dhimmel]

Sorry about my misunderstanding... I didn't realize Anaconda's recipes were all openly licensed and available at https://github.com/AnacondaRecipes. That's great. And of course, I'm grateful to Continuum for their continued investment in open source and community resources.

My intent wasn't to suggest "forking" away from Anaconda / Continuum. I was just curious if it was possible to install a distribution where all the packages (including conda) start out using the conda-forge channel? If this were the case, then my conda version would not have gotten downgraded, which is what caused the issue in this specific case. If that for some reason is considered hostile to Continuum, my bad... I don't fully understand the architecture and relationship here.

[djsutherland]

So, trying to figure out exactly what makes this fail:

• Start with a clean install of latest anaconda. .condarc has channels set only to defaults.
• This currently gives you conda 4.4.10; conda update conda updates that to 4.5.0.
• Make an environment: conda create -n hello ipython.
• Add conda-forge as preferred to defaults in .condarc.
• conda update conda downgrades conda to 4.3.
• source activate hello still works. conda install, conda create, etc all seem to work.

Do you know what you did differently from that that made everything break, @dhimmel? It might be that there's some easy fix to recover whatever's broken about your installation, and if this is something that's going to be happening to a lot of people we should figure out how to avoid it.

(Using defaults as preferred to conda-forge can make some things break, since unfortunately not everything is binary-compatible in the two channels.)

[msarahan]

This is a delicate issue, and it's one that the core maintainers of Conda-Forge will discuss with Anaconda at their meeting at AnacondaCon. It is technically very easy to create such a distribution. However, there are deeper ramifications. One of Anaconda's most important assets is the mindshare of the installed userbase. This mindshare is what leads enterprises to pay for our other products, which is what allows us to contribute back to the community. Efforts to create distributions that can detract from that mindshare can be counterproductive to our ability to contribute to the community. We hope to find a delicate balance that keeps an optimal arrangement between conda-forge and Anaconda.

As a corrollary to @dougalsutherland,

(Using defaults as preferred to conda-forge can make some things break, since unfortunately not everything is binary-compatible in the two channels.)

Hopefully that's not going to be true too much longer. Using the new compilers will bring us into binary compatibility. We'll also need to maintain that with library versions, of course, but it is an explicit long-term goal for both Anaconda and Conda-Forge.

[dhimmel]

@dougalsutherland here are the steps as best as I can recall:

1. Install anaconda on Ubuntu 18.04 via Anaconda3-5.1.0-Linux-x86_64.sh
2. Add the following line to .bashrc: source ~/anaconda3/etc/profile.d/conda.sh
3. Run conda activate to activate base.
4. Run conda update --all, which will update conda to 4.5.0 among many other things.
5. Run conda config --add channels conda-forge
6. Run conda update --all which switches conda to use conda-forge and downgrades it

Now when I opened a new terminal I would get an error because the ~/anaconda3/etc/profile.d/conda.sh file did not exist. Therefore I could not run conda activate. I also tried source activate which seemed to work one time perhaps. But then I think that stopped working. I also started getting errors because something expected the environment to be named root, but the name was base (or vice versa). Sorry that I didn't save the error messages.

[jakirkham]

There has been discussion about having a conda-forge based installer in issue ( conda-forge/conda-forge.github.io#90 ) and other associated issues. Thoughts on this have been primarily geared towards using an installer in the conda-forge infrastructure not as a user facing installer. Building an installer for general community use is certainly technically achievable with the tools around today.

What is less clear is how to design and provide such an installer with the interest of the whole community (which includes Anaconda) at heart. Solving such a problem would require a fair bit of discussion between those within and outside of Anaconda, who (to be explicit) are all members of conda-forge. For clarity, if such an installer were to be pushed, we would want Anaconda to be on board. This is a tricky issue for various reasons, but can understand it would not be obvious at first glance.

Also would highlight that developers at Anaconda have spent many long hours to design elegant solutions with conda-forge in particular in mind. All of this done in the open and with close collaboration of people inside and outside of Anaconda. The newer versions of things like conda, conda-build, and recent compilers carefully built with our needs taken into account offer significant advancement for the larger community both for package maintainers and users of our stacks. Needless to say we are very appreciative of all of this great work. Admittedly this migration to newer tools also highlight pain points at present (as discussed above), which you have unfortunately stumbled upon. We are generally interested in solving the pain points and bring the stacks closer together. This should only add value for everyone.

[djsutherland]

Ah – I haven't used new versions of conda (since I have conda-forge first in my channels), and use a custom zsh wrapper around conda envs, so I wasn't aware of the new activation stuff.

Okay, here's a way to reproduce it, and how to solve afterwards:

• docker run -it ubuntu bash to get in a clean, minimal Ubuntu 16.04 install
• apt-get update && apt-get install -y wget bzip2 because this is a minimal install
• wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
• bash Miniconda3-latest-Linux-x86_64.sh -b
• bash to enter a new subshell, so we can simulate opening a new shell within the same Docker environment
• source ~/miniconda3/etc/profile.d/conda.sh
• conda activate
• conda update conda (gets conda 4.5)
• conda update --all updates a few things
• conda config --add channels conda-forge
• conda update --all changes a bunch of things, including downgrading conda to 4.3.34. It eventually exits with code 2 after saying:

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
usage: conda [-h] [-V] command ...
conda: error: argument command: invalid choice: '/root/miniconda3/bin/conda' (choose from 'info', 'help', 'list', 'search', 'create', 'install', 'update', 'upgrade', 'remove', 'uninstall', 'config', 'clean', 'package')

• Now pretty much every conda command complains about

CondaEnvironmentNotFoundError: Could not find environment: /root/miniconda3/envs/base .
You can list all discoverable environments with `conda info --envs`.

• But! if I close this shell and start a new one (exit and then bash), and switch to the old style of activating things instead of the no-longer-extant source ~/anaconda3/etc/profile.d/conda.sh, then everything works again:
• export PATH=$HOME/miniconda3/bin:$PATH
• conda install ipython
• conda create -n tmp ipython; source activate tmp

So all you have to do to switch over to a fully conda-forge-ified install, until we can get new versions of conda up on conda-forge, is to replace the source line in your bashrc with a PATH-modifying line, and use source activate / source deactivate to switch environments instead of the new conda activate stuff.

[SylvainCorlay]

On the subject of having an official conda-forge-based miniconda, that does not require the default channel, it would make sense in my opinion, mostly because of the numerous binary compatibility issues that conda-forge and default have. This impacts packages sur as R and GDAL. Getting the conda-forge package to work requires carefully setting up an environment where almost nothing gets installed from default. Installing the conda-forge package in the root environment of a full anaconda distribution will just fail.

Now, regarding @mingwandroid's concern on whether this would be a problem to have such a split, I am not sure at all. After all, conda-forge could remain a reliable source of recipes, and the more healthy conda-forge is, the more reliable the recipes will be.

It will probably be increasingly difficult to have a continuously updated distribution from the same default channel, and even Anaconda Inc will probably need to version their distribution and make channels target a specific version, just like an Ubuntu package repository is meant for one specific version of ubuntu (artful, xenial, trusty) and PPAs have different flavors of the builds for each ubuntu version. If we end up in such a situation, with different distributions anaconda inc, we can totally have distributions from the community maintained conda-forge.

[djsutherland]

@sdmcclure We've had new versions of conda on conda-forge for months now, and indeed yours is 4.5.6, so whatever happened is different than what this thread is about....

[sdmcclure]

@dougalsutherland sorry, I've found the correct issue and a solution. I'll delete my comment.

thanks,

[MitraTj]

Solution

For others who are having this issue, I resolved it by switching the order of the channels in ~/.condarc to become:

channels:
  - defaults
  - conda-forge

Then I ran

~/anaconda3/bin/conda update --all

which made the following change to conda:

    conda:                              4.3.34-py36_0                 conda-forge --> 4.5.0-py36_0              

Question

I'm not an expert on conda, so most of the discussion in this issue is over my head. However, one thing that would have prevented my issue is if I could install miniconda/anaconda directly from conda-forge. As conda-forge grows, I think a lot of users (myself included) will want to interface only with the community driven & open source conda-forge rather than the partially-proprietary continuum anaconda. Would it make sense to have a conda-forge installer where the default channel is conda-forge? In other words, can you install miniconda and have that contain the conda-forge version of conda right off the bat?

Thank you.
However, I couldn't solve this problem.
Any other solution?

CondaHTTPError: HTTP 000 CONNECTION FAILED for url https://repo.anaconda.com/pkgs/main/noarch/repodata.json.bz2
Elapsed: -

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.

If your current network has https://www.anaconda.com blocked, please file
a support request with your network engineering team.

ConnectionError(MaxRetryError("HTTPSConnectionPool(host='repo.anaconda.com', port=443): Max retries exceeded with url: /pkgs/main/noarch/repodata.json.bz2 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f2eb3081128>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))",),)

A reportable application error has occurred. Conda has prepared the above report.

[DonAurelio]

I solve this problem installing the version 4.6.14 of conda as suggested on this issue 9004 before perform further installs.

conda install conda=4.6.14

It looks like further versions of conda are introducing this error.

[jakirkham]

Hi all, this issue is quite old and unlikely to be related to any issues today. Please open a new issue with a reproducer. Thanks!