sourcegraph · nicksnyder · Feb 11, 2019 · Feb 6, 2019 · Feb 6, 2019 · Feb 6, 2019
diff --git a/doc/dev/product/index.md b/doc/dev/product/index.md
@@ -1,6 +1,6 @@
 # Product
 
-This document is about *how* we do product at Sourcegraph. For the *what*, see the [product roadmap](../roadmap/index.md).
+This document is about *how* we plan product changes at Sourcegraph. For the *what*, see the [product roadmap](../roadmap/index.md).
 
 ## Goals
 
@@ -71,7 +71,9 @@ When we have relaxed this in the past, the results have been bad and the overwhe
 
 We receive tons of feature request and bug reports, more than we can handle. This means we must frequently say "no" or prioritize things less urgently than some people would like. Our job is to find the most important things to work on.
 
-## Product manager
+## Roles
+
+### Product manager
 
 The product manager is responsible for prioritization, which means ensuring the following is true:
 
@@ -89,3 +91,40 @@ The right mix of these 3 things depends on the team and what they're working on
 - For other teams, the PM will be the first to respond to and prioritize issues that are filed on the team.
 
 No matter the chosen mix, the PM is still responsible for prioritization. For example, if the PM doesn't effectively educate the devs so they can plan and triage well, then the PM needs to step up on planning and triage (as a backstop).
+
+### Tech lead
+
+Each project has one tech lead. A tech lead serves in that role for one or more release cycles.
+
+The tech lead is responsible for making sure the following two statements are true:
+
+> The issues for the current release milestone describe the problem/solution.
+>
+> The issues for the current release milestone are completed (closed) before the release.
+
+The following sections give more details.
+
+#### The issues for each release milestone are completed (closed) before the release.
+
+This means the tech lead needs to do 2 things:
+
+- **Estimate** at planning time what can get done in the time alotted for a release milestone.
+- **Reschedule** issues continuously into/from the milestone depending on the pace and [triage](../issues.md#triage).
+
+It's impossible to estimate perfectly, so the tech lead needs to continuously monitor the team's progress and make adjustments accordingly.
+
+If it looks like the team won't be able to complete all planned issues by the release date, the tech lead **must** communicate this in the #product channel and follow up with a combination of:
+
+- helping unblock or accelerate the work of individual teammates
+- descoping issues in this milestone
+- deferring issues to a later milestone
+- getting help from other people on certain issues
+
+The [product manager](#product-manager) can help here, especially with product and priority questions for descoping or deferring work.
+
+#### The issues for each release milestone describe the problem/solution.
+
+This means the tech lead needs to do 2 things:
+
+- Ensure the issues for feature work have enough information for the devs who are implementing the feature.
+- Gather information for bug reports (including documenting/automating how other people can supply the info needed to diagnose/fix the bug).
diff --git a/doc/dev/release_issue_template.md b/doc/dev/release_issue_template.md
@@ -0,0 +1,71 @@
+<!--
+This template is used for our monthly major/minor releases of Sourcegraph.
+It is not used for patch releases.
+-->
+
+# MAJOR.MINOR Release (YYYY-MM-DD)
+
+## No later than 5 working days before release (YYYY-MM-DD)
+
+- [ ] Choose dates/times for the steps in this release process and update this issue template accordingly. Note that this template references _working days_, which do not include weekends or holidays observed by Sourcegraph.
+- [ ] Send message to #dev-announce with a link to this tracking issue to notify the team of the release schedule.
+- [ ] Private message each teammate who has open issues in the milestone and ask them to remove any issues that won't be done three working days before the release.
+
+## 3 working days before release (YYYY-MM-DD)
+
+- [ ] **HH:MM AM/PM PT** Create the branch for this release off of master and tag the first release candidate (e.g. `v3.0.0-rc.1`).
+- [ ] Send a message to #dev-announce to announce the release candidate.
+- [ ] Run Sourcegraph Docker image with no previous data.
+    - [ ] Initialize the site by creating an admin account.
+    - [ ] Add a public repository (i.e. https://github.com/sourcegraph/sourcegraph).
+    - [ ] Add a private repository (i.e. https://github.com/sourcegraph/infrastructure).
+    - [ ] Verify that code search returns results as you expect (depending on the repositories that you added).
+    - [ ] Verify that code intelligence works as you expect.
+        - [ ] Go to definition works in Go.
+        - [ ] Go to definition works in TypeScript.
+        - [ ] Find references works in Go.
+        - [ ] Find references works in TypeScript.
+- [ ] Upgrade Sourcegraph Docker image from previous released version.
+    - [ ] Initialize the site by creating an admin account.
+    - [ ] Add a public repository (i.e. https://github.com/sourcegraph/sourcegraph).
+    - [ ] Add a private repository (i.e. https://github.com/sourcegraph/infrastructure).
+    - [ ] Verify that code search returns results as you expect (depending on the repositories that you added).
+    - [ ] Verify that code intelligence works as you expect.
+        - [ ] Go to definition works in Go.
+        - [ ] Go to definition works in TypeScript.
+        - [ ] Find references works in Go.
+        - [ ] Find references works in TypeScript.
+- [ ] Run Sourcegraph on a clean Kubernetes cluster with no previous data.
+    - [ ] Initialize the site by creating an admin account.
+    - [ ] Add a public repository (i.e. https://github.com/sourcegraph/sourcegraph).
+    - [ ] Add a private repository (i.e. https://github.com/sourcegraph/infrastructure).
+    - [ ] Verify that code search returns results as you expect (depending on the repositories that you added).
+    - [ ] Verify that code intelligence works as you expect.
+        - [ ] Go to definition works in Go.
+        - [ ] Go to definition works in TypeScript.
+        - [ ] Find references works in Go.
+        - [ ] Find references works in TypeScript.
+- [ ] Upgrade Sourcegraph on a Kubernetes cluster.
+    - [ ] Initialize the site by creating an admin account.
+    - [ ] Add a public repository (i.e. https://github.com/sourcegraph/sourcegraph).
+    - [ ] Add a private repository (i.e. https://github.com/sourcegraph/infrastructure).
+    - [ ] Verify that code search returns results as you expect (depending on the repositories that you added).
+    - [ ] Verify that code intelligence works as you expect.
+        - [ ] Go to definition works in Go.
+        - [ ] Go to definition works in TypeScript.
+        - [ ] Find references works in Go.
+        - [ ] Find references works in TypeScript.
+- [ ] Add any [release blocking](releases.md#blocking) issues as checklist items and start working to resolve them.
+
+## 1 working day before the release (YYYY-MM-DD)
+
+- [ ] **HH:MM AM/PM PT** Tag a final release candidate. Barring a new [release blocking](releases.md#blocking) issue, this will become the final release.
+- [ ] Send a message to #dev-announce to announce the release candidate.
+- [ ] Re-test any flows that might have been impacted by commits that have been `git cherry-picked` into the release branch.
+
+## On or before release day (YYYY-MM-DD)
+
+- [ ] **HH:MM AM/PM PT** Tag the final release and wait for the Docker images to get pushed (make sure you do this in advance of when you want to merge/publish the blog post).
+- [ ] **HH:MM AM/PM PT** Merge the blog post.
+- [ ] Review all issues in the release milestone. Backlog things that didn't make it into the release and ping issues that still need to be done for the release (e.g. Tweets, marketing).
+- [ ] Close this issue.
diff --git a/doc/dev/releases.md b/doc/dev/releases.md
@@ -1,83 +1,118 @@
 # Releases
 
-This document describes how we plan and ship releases at Sourcegraph.
+This document describes how we release Sourcegraph.
+
+## Goal
+
+The goal of our release process is to make releases boring, regular, and eventually, automated.
 
 ## Releases are monthly
 
-We ship a release on the 20th day of each month. (Except in February 2019, where we will ship on the 4th and 20th.) Why the 20th? Because it's a day number that we can hit each month (any other number would result in a December release that comes out too close to Christmas, or a January release that comes out too soon after New Year's Day).
+We release Sourcegraph by 10am PT on the 20th day of each month. No exception is made for weekends or holidays.
+
+"Release" means:
+- The Docker images are available for download.
+- The blog post is published.
 
 The release always ships on time, even if it's missing features or bug fixes we hoped to get in ([why?](https://about.gitlab.com/2015/12/07/why-we-shift-objectives-and-not-release-dates-at-gitlab/)).
 
-## Milestones and issues
+### Why the 20th?
 
-Each release has a GitHub milestone ([in all repositories](issues.md#multiple-repositories)) whose name is the version number, such as `3.1`:
+We don't want to ship a release too late in December because the Sourcegraph team has a scheduled break December 24 through January 1.
 
-- A release milestone's [issues](issues.md) constitute the release plan (i.e., the canonical definition of the work for the release).
-- A release is ready when its milestone has zero open issues ([across all repositories](issues.md#multiple-repositories)).
+### Why aren't releases continuous?
 
-The issues for a release come from two sources:
+Although [Sourcegraph.com](https://sourcegraph.com) is continuously deployed (from sourcegraph/sourcegraph@`master`), the version of Sourcegraph that customers use is not continuously released or updated. This is because:
 
-- [product planning](product/index.md#planning) (work planned for the release ahead of time)
-- [issue triage](issues.md#triage) (issues filed at any time)
+- We don't think customers would be comfortable with a continuously updated service running on their own infrastructure, for security and stability reasons.
+- We haven't built the automated testing and update infrastructure to make continuous customer releases reliable.
 
-### Process
+In the future, we may introduce continuous releases if these issues become surmountable.
 
-A release cycle is [1 month](#releases-are-monthly). Each release cycle goes as follows (where T is the start of the release cycle, beginning the moment when the previous release ships):
+## Release process
 
-- Between T-2 weeks and T: for all projects, [product planning](product/index.md#planning) for the release is done; tracking issues and all other foreseen work issues exist.
-  - Because [product planning is continuous](product/index.md#product-planning-is-continuous), these may have been ready even earlier for certain projects.
-- Between T and T+1 month:
-  - Each project team works on "making the tracking issue come true" by closing all project issues in the release milestone.
-  - [Triage issues](issues.md#triage) as they come in (which may require updates to the project plan for the current release because they supply us with new information).
-  - TODO: document testing and release preparation
-- T+1 month:
-  - Release! (TODO: document process and also how we do it if the 20th is on a non-work day)
-  - [Start a retrospective.](retrospectives/index.md#how-to-lead-a-retrospective)
+What is the process we follow to release?
 
-## Roles
+### Release captains
 
-### Tech lead
+The release captain is _responsible_ for managing the release process and ensuring that the release happens on time. The release captain may _delegate_ work to other teammates, but such delegation does not absolve the release captain of their responsibility to ensure that delegated work gets done.
 
-Each project has one tech lead. A tech lead serves in that role for one or more release cycles.
+No later than 5 _working days_ before the release day the release captain creates a tracking issue using the [release issue template](release_issue_template.md) and assigns it to themself to complete.
 
-The tech lead is responsible for making sure the following two statements are true:
+| Version | Captain | Release Date |
+|---------|---------|--------------|
+| 3.0 | @nicksnyder | 2019-02-07 (Wednesday) |
+| 3.1 | @nicksnyder | 2019-02-20 (Wednesday) |
+| 3.2 | @nicksnyder | 2019-03-20 (Wednesday) |
+| 3.3 | @beyang | 2019-04-20 (Saturday) |
+| 3.4 | @ggilmore | 2019-05-20 (Monday) |
+| 3.5 | @keegancsmith | 2019-06-20 (Thursday) |
+| 3.6 | @slimsag | 2019-07-20 (Saturday) |
+| 3.7 | @ijsnow | 2019-08-20 (Tuesday) |
+| 3.8 | @tsenart | 2019-09-20 (Friday) |
+| 3.9 | @lguychard | 2019-10-20 (Sunday) |
+| 3.10 | @attfarhan | 2019-11-20 (Wednesday) |
+| 3.11 | @chrismwendt | 2019-12-20 (Saturday) |
+| 3.12 | @vanesa | 2020-01-20 (Monday, MLK) |
+| 3.13 | @felixfbecker | 2020-02-20 (Thursday) |
 
-> The issues for the current release milestone are completed (closed) by the release.
->
-> The issues for the current release milestone describe the problem/solution.
+### Release branches
 
-The following sections give more details.
+Each release of [Sourcegraph](https://github.com/sourcegraph/sourcegraph) is tagged from a release branch.
+- Major releases, minor releases, and their release candidates (e.g. `v3.0.0-rc.1`, `v3.0.0`, `v3.1.0-rc.1`, `v3.1.0`) are tagged from the release branch (e.g. `3.0`, `3.1`) that has been branched off `master` for that release.
+- Patch releases and their release candidates (e.g. `v3.0.1-rc.1`, `v3.0.1`, `v3.1.1-rc.1`, `v3.1.1`) are tagged from the release branch of the corresponding major or minor release (e.g. `3.0`, `3.1`).
 
-#### The issues for each release milestone are completed (closed) by the release.
+To avoid confusion between tags and branches:
 
-This means the tech lead needs to do 2 things:
+- Tags are always the full semantic version with a leading `v` (e.g. `v2.10.0`)
+- Branches are always the dot-separated major/minor versions with no leading `v` (e.g. `2.10`).
 
-- **Estimate** at planning time how much can get done in the time alotted for a release milestone.
-- **Reschedule** issues continuously into/from the milestone depending on the pace and [triage](issues.md#triage).
+Development continues to happen on `master` during the release process.
 
-It's impossible to estimate perfectly, so the tech lead needs to continuously monitor the team's progress and make adjustments accordingly.
+#### Example
 
-If it looks like the team won't be able to complete the issues by the release, the tech lead **must** mention this in the #product channel and needs to do some combination of:
+Here is an example git commit history:
 
-- helping unblock or accelerate the work of individual teammates
-- descoping issues in this milestone
-- deferring issues to a later milestone
-- getting help from other people on certain issues
+1. The release captain creates the `3.0` release branch at commit `B`.
+1. The release captain tags the release candidate `v3.0.0-rc.1` at commit `B`.
+1. A feature is commited to `master` in commit `C`. It will not ship in `3.0`.
+1. An issue is found in the release candidate and a fix is committed to `master` in commit `D`.
+1. The release captain cherry picks `D` from `master` into `3.0`.
+1. The release captain tags `v3.0.0` on the `3.0` release branch.
+1. Development continues on master with commits `E`, `F`, `G`, `H`.
+1. Commit `F` fixes a critical bug that impacts 3.0, so it is cherry picked onto the `3.0` release branch and `v3.0.1` is tagged.
+1. The release captain (different person) for 3.1 creates the `3.1` release branch at commit `H` and a new release cycle begins.
+1. Commit `J` fixes a critical bug that impacts both 3.0 and 3.1, so it is cherry picked into both `3.0` and `3.1` release branches and new releases are tagged (`v3.0.2`, `v3.1.2`).
 
-The [product manager](product/index.md#product-manager) can help here, especially with product and priority questions for descoping or deferring work.
+```
+A---B---C---D---E---F---G---H---I---J---K---L (master branch)
+     \                       \                   
+      \                       `---v3.1.0-rc.1---I'---v3.1.0---J'---v3.1.2 (3.1 release branch)
+       \
+        `---v3.0.0-rc.1---D'---v3.0.0---F'---v3.0.1---J'---v3.0.2 (3.0 release branch)
+```
 
-#### The issues for each release milestone describe the problem/solution.
+### Issues
 
-This means the tech lead needs to do 2 things:
+How do we deal with issues that are found during the release process?
 
-- Ensure the issues for feature work have enough information for the devs who are implementing the feature.
-- Gather information for bug reports (including documenting/automating how other people can supply the info needed to diagnose/fix the bug).
+#### Blocking
 
-## We don't release continuously
+The release always ships on time, even if it's missing features or bug fixes we hoped to get in ([why?](https://about.gitlab.com/2015/12/07/why-we-shift-objectives-and-not-release-dates-at-gitlab/)).
 
-Although [Sourcegraph.com](https://sourcegraph.com) is continuously deployed (from sourcegraph/sourcegraph@`master`), the version of Sourcegraph that customers use is not continuously released or updated. This is because:
+There are only three kinds of issues that are eligible to block a release:
 
-- We don't think customers would be comfortable with a continuously updated service running on their own infrastructure, for security and stability reasons.
-- We haven't built the automated testing and update infrastructure to make continuous customer releases reliable.
+1. Issues that literally prevent us from tagging a release (i.e. our CI logic to produce builds from git tags is broken).
+2. Issues that break one of the critical workflows explicitly tested in the [release issue template](release_issue_template.md) which would impact _most_ customers and don't have workarounds. 
+3. Critical security _regressions_ from the previous release.
 
-In the future, we may introduce continuous releases if these issues become surmountable.
+Only the release captain can label something as release blocking.
+
+The release captain has unlimited power to make changes to the release branch to resolve release blocking issues, including but not limited to:
+- Fixing the issue in the release branch.
+- Reverting commits in the release branch.
+- Re-cutting the release branch from an older working commit off master.
+
+#### Non-blocking
+
+Most issues are non-blocking. Fixes to non-blocking issues can be fixed in `master` by the code owner who can then `git cherry-pick` those commits into the release branch with the approval of the release captain. Alternatively, broken features can be reverted out of the release branch or disabled via feature flags if they aren't ready or are too buggy.