Skip to content
This repository has been archived by the owner on Aug 28, 2020. It is now read-only.

Consider continuous deployment for this Repo #5

Closed
3 of 6 tasks
yuvipanda opened this issue Apr 2, 2018 · 15 comments
Closed
3 of 6 tasks

Consider continuous deployment for this Repo #5

yuvipanda opened this issue Apr 2, 2018 · 15 comments

Comments

@yuvipanda
Copy link
Member

yuvipanda commented Apr 2, 2018 •
edited by rabernat

This is awesome work, @jacobtomlinson! I'm also excited to see chartpress being used :)

What you think of continuous deployment for this repo? It took the pressure off me and other ops-y people in both mybinder.org and the berkeley jupyterhub deployment. See http://mybinder-sre.readthedocs.io/en/latest/deployment/how.html for how deployments to mybinder.org work. https://github.com/jupyterhub/mybinder.org-deploy/graphs/contributors shows that people of various skillsets can deploy with confidence.

This will also allow us (The JupyterHub community) to document good practices around continuous deployment, security management in public repos, and growing a community around your deployment.

Possible sequence of steps to doing CD:

  • Move the docker image contents here
  • Use git-crypt to store deployment secrets. MyBinder.org uses it to good effect!
  • Set up automatic build and push on merge for docker image
  • Create a 'staging' environment that mimics the production environment
  • Create a Google Cloud Service Account that has just enough rights to do a helm upgrade
  • Create two branches - 'staging' and 'prod'. Merging to 'staging' will deploy to the staging environment. Deployer then verifies staging environment works as intended, and merges PR to 'prod' environment after.

Open questions

  • Who has merge (and hence deploy) rights?
  • Whose responsibility it is to fix stuff when things break?
  • How do we incorporate other deployments (on AWS? etc) to this CD pipeline?
@yuvipanda
Copy link
Member Author

/cc @mrocklin who might be interested too.

@jacobtomlinson
Copy link
Member

I agree this would be a good thing to do. Thanks for the advice!

This repo is the generic helm chart for making 'a pangeo'. I wonder if you're thinking about pangeo.pydata.org which I would describe as 'an instance of a pangeo' and is managed from pangeo-data/pangeo. So perhaps the CI/CD should go there?

I would be keen to add better automated testing and publishing of charts to this repo.

@yuvipanda
Copy link
Member Author

yuvipanda commented Apr 3, 2018 via email

@jacobtomlinson
Copy link
Member

I'm keen for Pangeo to be a generic thing, like JupyterHub, which many people can deploy. pangeo.pydata.org is the reference deployment, but we also have our own at the Met Office.

Yeah I've got Travis set up to auto build and deploy using chartpress, but I set it up to only deploy on tags which means the commit range is not provided. I raised #2 to discuss solutions.

@yuvipanda
Copy link
Member Author

yuvipanda commented Apr 3, 2018 via email

@jacobtomlinson
Copy link
Member

jacobtomlinson commented Apr 3, 2018 •
edited

Tags are created with intent to mark a milestone, have been tested more thoroughly and come with some (possibly small) guarantees that it will definitely work. They also use semver which is well understood and means that package managers can intelligently upgrade.

I feel that using commits causes headaches because it is not always obvious which commit is the latest, particularly because they often get listed alphabetically. It is also hard to know where you revert back to if you encounter a bug, if there have been 30 commits since the last version you were on do you go back to where you were before to try somewhere in the middle.

As a first step I think using git describe --tags --abbrev=1 which includes the number of commits since the last tag helps make things clearer.

@rabernat
Copy link
Member

rabernat commented Apr 3, 2018

Thank you both for your work on this!

For me, continuous deployment of pangeo.pydata.org is a very high priority (see e.g. pangeo-data/pangeo#131). The reason is that we really don't have the resources for a full time sys-admin team to babysit the cluster. Right now, the burden of pushing bugfix changes and redeploying the cluster falls on @mrocklin, @jacobtomlinson, or anyone else with the tech savvy who happens to be available when a meltdown occurs. This is not really sustainable.

I do also agree with Jacob that we want to provide a "generic" pangeo deployment template, with pangeo.pydata.org as a particular instance.

@rabernat
Copy link
Member

Just following up on this older discussion, now that I have gotten my feet wet a bit with docker, helm, etc.

In my rush to get a new chart built, I just enabled deployment of every commit, contrary to @jacobtomlinson's recommendations above. I am happy to roll this back, as now I understand the arguments better. However, it seems like jupyterhub deploys every commit, so clearly there are different approaches possible.

My main question is: given that this is the "generic" pangeo helm chart, what additional configuration is required for a specific deployment (e.g. pangeo.pydata.org, met office, etc.) And where should that config live? Should we have an additional repo dedicated just to pangeo.pydata.org deployment?

@rabernat
Copy link
Member

We continue to update the docker files in the main pangeo repo (pangeo-data/pangeo#261). I would really like to get the docker files moved here and automatically built.

@yuvipanda would you care to suggest how we might move towards implementing the continuous deployment as described in your original post? Just start checking off the boxes one by one?

@rabernat rabernat mentioned this issue May 19, 2018
Merged
@rabernat rabernat mentioned this issue Jun 14, 2018
Closed
@jgerardsimcock
Copy link

What is the procedure for having travis deploy charts to gh-pages via chartpress? My .travis.yml and deploy.sh files are similar to those in this repo. Its not clear to me how to configure the gh-pages branch for travis to do what it needs to do. Travis is choking on this line in chartpress.py

@jacobtomlinson
Copy link
Member

@jgerardsimcock you also need to have an SSH key which can push to the gh-pages branch on your repo. We are doing this with an encrypted secret file. You also need to start off with an orphaned gh-pages branch.

Shout if you need any more help.

@rabernat
Copy link
Member

We have accomplished the first three steps of @yuvipanda's checklist.

But issue #54 highlights the need for a test cluster before we deploy to the production cluster. Maybe we can work on this this week at the Pangeo meeting.

@yuvipanda
Copy link
Member Author

https://github.com/yuvipanda/hubploy is the code I'm working on that lets you do CD easily for things like this. It complements chartpress. Some docs here: https://hubploy.readthedocs.io/ but more coming.

@guillaumeeb
Copy link
Member

I'm reviving some old issue here, but I'd like to revive this repo. So sorry for the noise in advance.

I think this one can be closed as this as been done in https://github.com/pangeo-data/dev.pangeo.io-deploy.

I support the distinction of @jacobtomlinson between 'A pangeo' and 'The Pangeo'. So keeping this repo without CD on a given site seems fine.

@jhamman
Copy link
Member

jhamman commented Jan 26, 2019

I'm in agreement with @guillaumeeb here. I will close this in favor of the recent hubploy deployments.

@jhamman jhamman closed this as completed Jan 26, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@yuvipanda @rabernat @jacobtomlinson @jhamman @jgerardsimcock @guillaumeeb