Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

freeze environment in hub image #1562

Merged
merged 14 commits into from
Apr 8, 2020
Merged

Conversation

minrk
Copy link
Member

@minrk minrk commented Jan 31, 2020

separates our 'loose' dependencies that we specify from the pinned environment.

Adds a dependencies script to the hub image, which wraps running pip-compile and pip list --outdated in the image to try to automate the process of keeping these things up-to-date.

How it works:

  • builds the hub image, taking build arguments from chartpress config, installing from requirements.txt
  • requirements.in contains our human-edited dependencies
  • requirements.txt contains a full, frozen environment, autogenerated by pi
  • ./dependencies freeze runs pip-compile in the final image to generate requirements.txt. This ensures the environment is the same without maintaining a separate image, at the expense of building the image more often than is strictly necessary
  • ./dependencies outdated runs pip list --outdated in the final image (after installing from the current requirements.txt) and gives a report of what packages are out-of-date

For example, running ./dependencies outdated immediately after writing it gave me this output:

Checking for outdated dependencies with pip.
Have jupyterhub-firstuseauthenticator==0.12, latest is jupyterhub-firstuseauthenticator==0.13.0
There are outdated dependencies!
To pick up any versions outside the range(s) specified in requirements.in,
update that file.
To update the environment, run `./dependencies freeze`

I updated requirements.in to have jupyterhub-firstuseauthenticator==0.13.*
and re-ran ./dependencies freeze, which produced ff9873b

TODO:

  • test/discuss design/API/workflow
  • promote workflow to other images if we like it, either in this PR or after merging and learning from use on this one image for a while
  • consider removing most pinning in requirements.in, since for ~everything there we just want 'latest'. The only known pin-down that I think we really want there is kubernetes, which we want to update pretty conservatively and explicitly.
  • Maybe allow upgrades of individual packages more explicitly with pip-compile --upgrade-package X?
    This only works within the ranges pinned in requirements.in, so suggests that we remove most pins there.

@consideRatio consideRatio self-requested a review January 31, 2020 11:40
and more docstrings
@minrk
Copy link
Member Author

minrk commented Feb 1, 2020

Updates with last commit, after learning a bit about pip-compile. Now that requirements.txt exists, we must run pip-compile --upgrade to upgrade all dependencies or pip-compile --upgrade-package jupyterhub-kubespawner to upgrade one or more specific packages. These options are now passed through by freeze, so the workflow would be:

  1. ./dependencies outdated to check for updates, and
  2. if ranges should be changed edit requirements.in
  3. run one (or more) of:
    1. ./dependencies freeze with no args, should only make changes if requirements.in is still satisfied
    2. ./dependencies freeze --upgrade to upgrade all dependencies within pinnings of requirements.in
    3. ./dependencies freeze --upgrade-package jupyterhub-kubespawner to upgrade specific packages within pinnings

since all dependencies are actually pinned in requirements.txt until the freeze command updates them

only kubernetes is left pinned, maybe even that shouldn't be?
no changes to environment at all, just the comment about why cryptography is installed
Copy link
Member

@consideRatio consideRatio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent work! I've considered:

  1. Renaming of some variables to be more descriptive
  2. More descriptive docstrings
  3. To perhaps treat packages that are outdated differently if they are not part of requirements.in and/or requirements.txt when reporting about them. Also, like you comment, what to do about updates when they are outside the bound of the pinning within requirements.in? They may still have an update within the bound, so we are left not knowing for sure what goes on.

images/hub/dependencies Outdated Show resolved Hide resolved
images/hub/dependencies Outdated Show resolved Hide resolved
images/hub/dependencies Outdated Show resolved Hide resolved
images/hub/dependencies Show resolved Hide resolved
images/hub/dependencies Outdated Show resolved Hide resolved
images/hub/requirements.txt Outdated Show resolved Hide resolved
images/hub/dependencies Outdated Show resolved Hide resolved
minrk and others added 6 commits March 12, 2020 15:42
- specify custom compile command
- more detailed docstrings
- avoid using image to mean too many things
make it conditional on building in the dependencies wrapper
updates all dependencies
@consideRatio
Copy link
Member

Builds fails due to sphinx 3.0.0 is out and that have introduced an issue for circleci.

@consideRatio consideRatio merged commit 6d361f2 into jupyterhub:master Apr 8, 2020
@consideRatio
Copy link
Member

@minrk I tried this out and I love it, this is brilliant!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants