More structured deployment process

[yuvipanda]

Heya! As our team gets bigger, we should scale by making it easier and more painless to do deploys in quick and reliable ways, through a combination of automation & expectations. Here's a straw proposal!

Whenever a change is merged into binderhub, a PR is automatically made bumping chart version in this repo. It is the responsibility of the person merging the PR in binderhub to also merge the PR in this repo. This has a few advantages:

1. Increases bus factor. Since folks can't self-merge in binderhub (or repo2docker), but can here, this ensures that the person doing the deploy isn't the same person as the person who wrote the code. This spreads knowledge around and forces better docs / explanations.
2. Forces our deploy process to be super smooth & reliable. Currently our deploys almost always fail the first time, which is very bad form. We can 'get away with it' because the folks doing deploy have this unwritten knowledge to 'just restart it', which is very bad. If there's a larger number of folks doing deploys, it forces our process to be at the least easily documented, and our automations to be better.
3. It helps prevent the slide into hero culture, where only a few people are able to deploy big changes because deploying big changes is under tested, under documented & under automated. This is bad for the culture of the community & for the 'hero's too, and we need to explicitly ward this off before it begins.
4. Gives contributors to binderhub / repo2docker a 'instant gratification' boost - as soon as their PR is merged, it's live! No waiting for an indeterminate amount of time for people with 'deploy rights' to do a sacred act!

This doesn't mean we force people who want to merge PRs in those two repos to know how to deploy. Rather, this means we build enough reliable automation that they would be willing to and find it pleasurable enough to do a deploy when they merge the PR. Lots of automation work, but that's the only way we can scale something like this to many many people. This issue is just a place to gather consensus that this is something we want to move towards, so we can focus on the automation bits more.

Note that this is also just a straw proposal based on my personal opinions the other large software community I've been part of that also had to do deploys consistently (Wikimedia). Feel free to bring up objections or alternative proposals!

[yuvipanda]

/cc @willingc @betatim @choldgraf @minrk @Carreau @mariusvniekerk

[minrk]

👍 to this, and a first prototype is in progress on binderhub: jupyterhub/binderhub#222

The only possible downside I can see is an incentive to avoid merging a PR when one doesn't have the time to deal with the deploy (i.e. if I can't deploy right now, I also can't merge upstream PRs). I'm not sure that will be an issue, and it certainly doesn't mean we shouldn't give it a try.

[yuvipanda]

@minrk cool! IMO not merging a PR when one doesn't have time to deploy right now is a good thing, rather than bad thing. Merging and not deploying sortof blocks other people from merging and deploying, since they now become responsible for your PR too.

[minrk]

Makes sense!

[willingc]

I think it's a reasonable workflow to approve the code after review. If there's not time / support at the moment to merge, there's no harm done by a second review by someone else who can then merge. We're pretty fluid in terms of merging in a timely manner.

[choldgraf]

I'm +100 on this idea, the "expect a failure on first deploy" is getting really annoying :-)

[betatim]

I like the proposal.

One thing that I would love to see is that we could deploy to staging when we open the PR. The workflow of open PR -> instantly merge -> revert because failure ... not sure how we'd organise for that but yeah it would be nice and feel more in line with how the rest of GitHub works.

[choldgraf]

We are currently planning on short-term dealing with this by having planned deploys with team-members present. Here's another +100 on this issue in case somebody has the cycles to automate a binderhub/repo2docker bump!

[yuvipanda]

A lot has happened since this issue was filed, and I think we're in a good place now.