Releases
We appoint a release manager from the team, and rotate every month to share the load here (includes devs + em).
On release day, release manager:
- Checks for changes to the dependent functional tests that live in the server-syncstorage repo repo. If the tests have changed, we’ll need to create a new server-syncstorage release and manually specify that tag in the jenkins UI in order to trigger manual tests.
- Follows these instructions to create a new release as needed: both in the codebase, and as an official GitHub release.
- Files a new deployment request.
- Monitors production dashboards here and here during and after the release.
We have an assigned release manager, which will rotate monthly. For regular feature and maintenance work and non critical bug fixes, we follow a predictable 2 week release cycle. This means we have 2 weeks to test new changes against staging before they roll out to production.
- Releases happen every other Tuesday.
- Release calendar is here (internal link).
- If it’s a holiday/etc we’ll use the release calendar to communicate agreed upon changes accordingly.
- Staging:
- If new changes have merged to master since last release, the release manager creates a new release. The release gets automatically deployed to staging.
- If no new changes have merged to master since last release: no new release is created.
- Production:
- If the version in production does not match the version in staging (sometimes it will match if no release was created in the previous cycle), then the release manager files a new deployment request for Ops to manually release to production. This bug is a good example of what should be included in this request - ie, list of changes, required config changes, etc.
Here's an example to illustrate how this process works:
| Staging | Production | |
|---|---|---|
| Week 1 | v0.0.4 | v0.0.3 |
| Week 2 | v0.0.4 | v0.0.3 |
| Week 3 | v0.0.5 | v0.0.4 |
Most of the time, waiting 4 weeks to roll out a change to production is fine. Sometimes however, we’ll run into something that’s time sensitive and needs to be deployed more quickly. If that’s the case, the release manager can tag a new release at any time, and then follow the steps to request a production or staging deployment.
We’ll track these types of requests [here](Most of the time, waiting 4 weeks to roll out a change to production is fine. Sometimes however, we’ll run into something that’s time sensitive and needs to be deployed more quickly. If that’s the case, the release manager can tag a new release at any time, and then follow the steps to request a production or staging deployment.
We’ll track these types of requests here, in order to understand how often we decide to short circuit the regular release process.
Here’s what happens during the release cycle for these kinds of fixes: ), in order to understand how often we decide to short circuit the regular release process.
Here’s what happens during the release cycle for these kinds of fixes:
| Week 1 - staging | Week 1 - prod | Week 2 - staging | Week 2 - prod | |
|---|---|---|---|---|
| Staging fix ("warmfix") | If we can get a fix in, deploy it to staging. No more changes to the release process after that. | N/A | Staging fix replaces planned staging release for this week. | N/A |
| Production fix ("hotfix") | Hotfix is also deployed to staging (ie, we still want manual testing/etc) | Hotfix deployed. No more changes to the release process after that. | Hotfix deployed. No more changes to the release process after that. | Hotfix replaces planned prod release for this week. |
| Staging | Production | |
|---|---|---|
| Automated unit & integration tests pass | X | X |
| QA has not identified blockers via manual testing | X | |
| Release manager has determined we’re safe to release | X | X |
| Release date is not a holiday (if so, we’ll either skip for the week, or move back a day - will be communicated to relevant teams as needed) | X | X |
NEW GITHUB RELEASE CREATED -> Staging. When a new release is created, CircleCI is configured to automatically create a new docker tag and deploy to staging.
RELEASE MANAGER FILES OPS REQUEST -> Production. Following our regular release calendar, the release manager files a new deployment request for Ops to manually release to production as needed.