Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Bump conda version #637

Merged
merged 5 commits into from Apr 30, 2019

Conversation

@jhamman
Copy link
Contributor

jhamman commented Apr 4, 2019

This bumps the miniconda version to 4.5.12 (latest) and the conda version to 4.6.11 (latest). Conda 4.6 included a number of performance improvements that should help speed up install times for complex environments. It also included some improvements to conda's interoperability with pip. An overview of the conda release is available here: https://www.anaconda.com/conda-4-6-release/.

Fixes #593

@betatim

This comment has been minimized.

Copy link
Member

betatim commented Apr 4, 2019

Thanks for helping get this done! I edited your comment to include the issue that has been tracking this so it gets closed.

We should also refreeze our environments per #593 (comment)

What is your thinking behind bumping conda but not miniconda? My expectation was that we'd keep them in sync unless there is a reason not to.

@betatim

This comment has been minimized.

Copy link
Member

betatim commented Apr 4, 2019

The answer to "why not also update miniconda" is that there isn't a miniconda 4.6! See https://repo.anaconda.com/miniconda/

@betatim

This comment has been minimized.

Copy link
Member

betatim commented Apr 4, 2019

I think the TL;DR is that the new conda requires a Python version newer than 3.5. So we need to think of a way to deal with that.

I think the conda/simple-py2/verify::build FAILED failure is due to a network issue and will resolve itself. That's why the "verify" part of that test worked even though the build part had just previously failed.

@betatim betatim changed the title Bump conda version [WIP] Bump conda version Apr 8, 2019
@yuvipanda yuvipanda referenced this pull request Apr 12, 2019
@betatim

This comment has been minimized.

Copy link
Member

betatim commented Apr 13, 2019

The new version of conda doesn't work with Python 3.5 any more. At least I can't get it installed in a conda env with Python 3.5. First thought this was because conda-forge isn't building new packages for 3.5 any more but even without using conda-forge this seems to happen. There is nothing in the conda change log about this so I am suspicious/think I've made a mistake somewhere.

I think the way forward is that we need to start treating Python 3.5 separately. It already has a "deep frozen" environment.yml to setup the basic env. I am wondering if it is possible to adopt the same method we use for Python 2: have a "user env" and a "server env". The server env is always a modern Python 3, independent of what the user requested. Would it work to install conda there and use it in the Pythonn 3.5 env?

@yuvipanda

This comment has been minimized.

Copy link
Collaborator

yuvipanda commented Apr 15, 2019

@betatim I agree re: separate server / user env, (or 'notebook' env and 'kernel' env) similar to python2. It'll also help with stuff like #438

@betatim

This comment has been minimized.

Copy link
Member

betatim commented Apr 15, 2019

After the pointer to the Jupyter lab issue I think we will want to have separate user and kernel envs for everything. Right now we do it as a special case for Python 2. Do you think that is a good goal?

I think the downsides include: harder for users to install server extensions, impossible to organise for repositories that currently install server extensions to continue working, bigger docker images.

Advantages include: we don't have to special case certain setups, harder for users to break their kernel/notebook server

Is my assumption that notebook server extensions would need installing in the kernel env correct?

Overall: we have a few tricky trade-offs ahead of us. Is there a way to upgrade conda (resolve this PR) without having to first also have a decision on all the trade-offs? Maybe we spend a bit of extra effort to do so but I think this particular issue is worth resolving "now" instead of in a few weeks, even if it costs a bit extra in engineering effort.

@jhamman

This comment has been minimized.

Copy link
Contributor Author

jhamman commented Apr 15, 2019

I'm going to defer to you all on how to proceed here. FWIW, a refactor of the user/kernel envs is probably beyond what I can help with right now. If it makes sense to put this on hold until that is addressed, fine by me.

@yuvipanda

This comment has been minimized.

Copy link
Collaborator

yuvipanda commented Apr 15, 2019

@betatim yeah, if we have a 'notebook' env where notebook lives and a 'kernel env' where kernel lives, notebook extensions need to be in the notebook env.

We 'hack' this with python2 by having a 'requirements3.txt' that installs things into the notebook env. This lets people install notebook extensions.

Doing that by default for everything would definitely break lots of current repos that install server extensions. I'd say for now we special case 3.4 and 3.5, and figure out a better solution for this after.

@jhamman

This comment has been minimized.

Copy link
Contributor Author

jhamman commented Apr 15, 2019

Out of curiosity, do we know how ofter 3.4 and 3.5 are being used with repo2docker?

@minrk

This comment has been minimized.

Copy link
Member

minrk commented Apr 25, 2019

do we know how ofter 3.4 and 3.5 are being used with repo2docker?

We don't, but I'm guessing it's rather few. #651 should get this unstuck because it leaves conda alone in the root env, installing the notebook in a dedicated env. This will presumably come at some cost to image size from the extra env, but may also improve installation time since it removes the step where the root env is switched to the chosen Python version.

@betatim betatim merged commit f9620ed into jupyter:master Apr 30, 2019
2 of 3 checks passed
2 of 3 checks passed
continuous-integration/travis-ci/pr The Travis CI build failed
Details
ci/circleci: build_docs Your tests passed on CircleCI!
Details
ci/dockercloud Your tests passed in Docker Cloud
Details
@jhamman

This comment has been minimized.

Copy link
Contributor Author

jhamman commented Apr 30, 2019

Thanks all!

@betatim

This comment has been minimized.

Copy link
Member

betatim commented Apr 30, 2019

What GitHub magic is this that this PR also got merged when we merged #651? This is amazing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.