[WIP] Bump conda version

[jhamman]

This bumps the miniconda version to 4.5.12 (latest) and the conda version to 4.6.11 (latest). Conda 4.6 included a number of performance improvements that should help speed up install times for complex environments. It also included some improvements to conda's interoperability with pip. An overview of the conda release is available here: https://www.anaconda.com/conda-4-6-release/.

Fixes #593

[betatim]

Thanks for helping get this done! I edited your comment to include the issue that has been tracking this so it gets closed.

We should also refreeze our environments per #593 (comment)

What is your thinking behind bumping conda but not miniconda? My expectation was that we'd keep them in sync unless there is a reason not to.

[betatim]

The answer to "why not also update miniconda" is that there isn't a miniconda 4.6! See https://repo.anaconda.com/miniconda/

[betatim]

I think the TL;DR is that the new conda requires a Python version newer than 3.5. So we need to think of a way to deal with that.

I think the conda/simple-py2/verify::build FAILED failure is due to a network issue and will resolve itself. That's why the "verify" part of that test worked even though the build part had just previously failed.

[betatim]

The new version of conda doesn't work with Python 3.5 any more. At least I can't get it installed in a conda env with Python 3.5. First thought this was because conda-forge isn't building new packages for 3.5 any more but even without using conda-forge this seems to happen. There is nothing in the conda change log about this so I am suspicious/think I've made a mistake somewhere.

I think the way forward is that we need to start treating Python 3.5 separately. It already has a "deep frozen" environment.yml to setup the basic env. I am wondering if it is possible to adopt the same method we use for Python 2: have a "user env" and a "server env". The server env is always a modern Python 3, independent of what the user requested. Would it work to install conda there and use it in the Pythonn 3.5 env?

[yuvipanda]

@betatim I agree re: separate server / user env, (or 'notebook' env and 'kernel' env) similar to python2. It'll also help with stuff like #438

[betatim]

After the pointer to the Jupyter lab issue I think we will want to have separate user and kernel envs for everything. Right now we do it as a special case for Python 2. Do you think that is a good goal?

I think the downsides include: harder for users to install server extensions, impossible to organise for repositories that currently install server extensions to continue working, bigger docker images.

Advantages include: we don't have to special case certain setups, harder for users to break their kernel/notebook server

Is my assumption that notebook server extensions would need installing in the kernel env correct?

Overall: we have a few tricky trade-offs ahead of us. Is there a way to upgrade conda (resolve this PR) without having to first also have a decision on all the trade-offs? Maybe we spend a bit of extra effort to do so but I think this particular issue is worth resolving "now" instead of in a few weeks, even if it costs a bit extra in engineering effort.

[jhamman]

I'm going to defer to you all on how to proceed here. FWIW, a refactor of the user/kernel envs is probably beyond what I can help with right now. If it makes sense to put this on hold until that is addressed, fine by me.

[yuvipanda]

@betatim yeah, if we have a 'notebook' env where notebook lives and a 'kernel env' where kernel lives, notebook extensions need to be in the notebook env.

We 'hack' this with python2 by having a 'requirements3.txt' that installs things into the notebook env. This lets people install notebook extensions.

Doing that by default for everything would definitely break lots of current repos that install server extensions. I'd say for now we special case 3.4 and 3.5, and figure out a better solution for this after.

[jhamman]

Out of curiosity, do we know how ofter 3.4 and 3.5 are being used with repo2docker?

[minrk]

do we know how ofter 3.4 and 3.5 are being used with repo2docker?

We don't, but I'm guessing it's rather few. #651 should get this unstuck because it leaves conda alone in the root env, installing the notebook in a dedicated env. This will presumably come at some cost to image size from the extra env, but may also improve installation time since it removes the step where the root env is switched to the chosen Python version.

[jhamman]

Thanks all!

[betatim]

What GitHub magic is this that this PR also got merged when we merged #651? This is amazing.