Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Update notebook to latest release #475

Merged
merged 5 commits into from Dec 12, 2018

Conversation

@betatim
Copy link
Member

betatim commented Nov 19, 2018

This PR does the following:

  • upgrades the notebook to 5.7.2
  • uses pip to install the notebook in the Python 3.5 environment, and
  • unpins the tornado dependency.

There are no more new conda-forge packages for Python 3.5 so we have to switch to using to normal pip packages if we need to upgrade something.

tornado was pinned because v5 was new at the time and there were some hiccups. Unpinning now so we don't get stuck in 4.x land.

Not dealt with in this PR: jinja2 contains syntax errors when you run it on Python3.5 (yield in an async function). Need to check if they dropped support for 3.5 and haven't yet specified it in their requirements or if this is a bug.

@betatim betatim force-pushed the betatim:bump-notebook branch from c64c860 to 9cb2213 Nov 20, 2018
@betatim

This comment has been minimized.

Copy link
Member Author

betatim commented Nov 20, 2018

Currently exporting an env that has packages in it that have been installed with pip does not work properly. For example the Python 3.5 env export doesn't have notebook==5.7.2 in it. Packages that weren't also/first installed by conda do appear (nteract_on_jupyter)

Apparently this is because something in conda broke when using pip 18.

Should we handcraft/fix the frozen environment? Wait for conda 45.6 that contains a fix? Other ideas?

now that it's not being updated anymore

only security fixes from now on for Python 3.5
@minrk

This comment has been minimized.

Copy link
Member

minrk commented Dec 11, 2018

@betatim are you referring to this message?

pyc file failed to compile successfully
  python_exe_full_path: /opt/conda/envs/r2d/bin/python3.5
  py_full_path: /opt/conda/envs/r2d/lib/python3.5/site-packages/jinja2/asyncsupport.py
  pyc_full_path: /opt/conda/envs/r2d/lib/python3.5/site-packages/jinja2/__pycache__/asyncsupport.cpython-35.pyc
  compile rc: 1
  compile stdout:
  compile stderr:   File "/opt/conda/envs/r2d/lib/python3.5/site-packages/jinja2/asyncsupport.py", line 35
    yield event
    ^
SyntaxError: 'yield' inside async function

?

If so, that's .py->.pyc bytecompilation. It shouldn't be an issue because presumably the file isn't used on Python 3.5. If it's not imported, that error (which is logged but doesn't stop progress) doesn't matter.

Should we handcraft/fix the frozen environment?

Yes, I think now that Python 3.5 isn't building anymore, we should stop regenerating the frozen Python 3.5 environment. I've updated freeze.py to check for the GENERATED header in environment.frozen.yml so it won't regenerate a frozen env without that header, and removed that from the Python 3.5 env, and ensured that notebook 5.7.2 is installed with pip.

In the future, I think we should probably not rely on conda to install things with pip, and instead do the whole thing in two stages:

  1. conda install && conda env export only conda packages
  2. pip install && pip freeze requrements.txt

That should be a lot more robust, I think.

At some point, old Python 3 is also going to go the way of Python 2, and be supported only for the kernel env.

@betatim

This comment has been minimized.

Copy link
Member Author

betatim commented Dec 11, 2018

If so, that's .py->.pyc bytecompilation. It shouldn't be an issue because presumably the file isn't used on Python 3.5. If it's not imported, that error (which is logged but doesn't stop progress) doesn't matter.

That was my impression as well. I started looking into it mostly because it creates a lot of noise in the terminal that makes me ignore all the output which might contain errors that are actual errors. However I've made no progress on figuring out what the real fix is. Agreed that this seems to be harmless because it is unused.

Yes, I think now that Python 3.5 isn't building anymore we should stop regenerating the frozen Python 3.5 environment.

Do you mean the Python developers are not changing Python 3.5 any more or there are no more releases of the notebook for Python 3.5?

I like the idea of deep freezing (is that what we are calling it?) certain Python versions like you did here.

@betatim betatim referenced this pull request Dec 11, 2018
@minrk

This comment has been minimized.

Copy link
Member

minrk commented Dec 11, 2018

However I've made no progress on figuring out what the real fix is

There is nothing to fix. Python can bytecompile all the files in a package at install time, and happily moves on past any files it can't compile, noting which ones it couldn't. If these files aren't imported on Pythons where it can't be, then there's no issue.

Do you mean the Python developers are not changing Python 3.5 any more or there are no more releases of the notebook for Python 3.5?

I mainly mean relevant to us, that conda folks aren't building Python 3.5 packages anymore. Python 3.5 is no longer accepting bugfixes, but is accepting security fixes until 2020.

@betatim

This comment has been minimized.

Copy link
Member Author

betatim commented Dec 11, 2018

I vote to merge this once travis is happy. (The GH UI doesn't let me approve my own PR :) )

betatim added 2 commits Dec 11, 2018
The frozen Python 3.6 environment was empty so refreezing everything.
Added the generated environment.py-3.6.yml which used to be excluded by
gitignore because it was the only one not also commited to the repo.
@betatim betatim merged commit 52855c2 into jupyter:master Dec 12, 2018
4 checks passed
4 checks passed
ci/circleci: build_docs Your tests passed on CircleCI!
Details
codecov/patch 0% of diff hit (target 0%)
Details
codecov/project 75.18% (target 0%)
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@betatim betatim deleted the betatim:bump-notebook branch Dec 12, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.