Releng for v3.0

[severinrudie]

EDIT

For 3.0, due to an ongoing experiment we're collecting data for, we're not going to release until 12/3.

Overview

Soft code freeze for a release is the second Wednesday of the sprint, and most of releng happens between Wednesday and Friday, String freeze is the first Tuesday of the sprint.

Monday	Tuesday	Wednesday	Thursday	Friday
	String Freeze
		Soft Code Freeze	QA	Hard Code Freeze/Localization Due/Planning
	Release to Play Store

Requirements

• Jira account
• Bugzilla account
• Google Play access (for reviewing crashes)

Release Checklist

We will refer to the release that is going out as the current release.

Start of sprint [Monday, 1st week of sprint]

• Create milestone for upcoming release. (e.g. if you are doing releng for v2.2, create v2.3 milestone)
• If the upcoming release is a major (x.0) release, create an issue in the upcoming milestone: "What's New Entry for [upcoming release]" to track work for the SUMO page and Google Play release notes, e.g., if the current release is 2.3 but the upcoming one will be 3.0, make a "What's New" issue for 3.0. Product will use this to remember to check in with SUMO.
• Create an issue in the upcoming milestone: "Releng for v[release]". Find an engineer who will handle the next releng task and assign them.

String freeze [Tuesday, 1st week of sprint]

• Make sure all issues with label needs:strings and eng:ready have strings provided in comment 0. If there are no strings, ping the Content team in #fenix-team channel to provide them. (Content is @jpfaller, or @mheubusch)
• Open a PR with all new strings in the needs:strings label that have not yet been merged to master
• For each string included in the PR, update its issue's strings:needed labels from the PR to approved:strings

String freeze follow-up [Wednesday, 1st week of sprint]

• For any strings still labeled needs:strings
  • If strings have been provided by content team, open a PR with the newly provided strings. Update their strings:needed labels to approved:strings
  • Otherwise, escalate to Product because this may miss the release.

Soft code freeze [Wednesday, 2nd week of sprint]

• Merge any remaining automated L10N string PRs.

• Create a branch off of master (DO NOT PUSH YET) for the current milestone of format releases/v2.3 (where 2.3 is the current milestone). After that, anything landing in master will be part of the next milestone.

• On the Release branch, pin the AC version to the stable version (example) with commit message "Issue #<this releng issue>: Pin to stable AC <version> for release v2.3" (replacing 2.3 with the version)

• Go through the list of issues closed during this sprint and make sure they all have the correct milestone.

• Add eng:qa:needed flags on each issue that still needs it.

• Note: You will need code review to make changes to the release branch after this point, because it is a protected branch.

  • Push the branch.

• On GitHub, draft a GitHub Release with with a tag of the format v2.3.0-rc.1 with the release branch as the target. Check the pre-release checkbox. This will kick off a build of the branch. You can see it in the mouseover of the CI badge of the branch in the commits view.

  • If you need to trigger a new RC build, you will need to draft and publish a new (pre-release) release. Editing an existing release and creating a new tag will not trigger a new RC build.

• Upload the APK from the Taskcluster signing task to the releases page.

• Create a new PI (product integrity) request in Jira. You can clone this issue.

SUMO Verification [Thursday, 2nd week of sprint]

• If the current release is a major (x.0) release, review the SUMO article contents of the whats new / other sumo pages and make sure they are accurate with what is in this release. If not, escalate to Product Owner.

During Product Integrity (Soft code freeze until PI green signoff)

• Check Google Play for new crashes. File issues and triage.
• If bugs are considered release blocker then find someone to fix them on master and the milestone branch (cherry-pick / uplift)
• If needed tag a new RC version (e.g. v1.0-RC2) and follow the submission checklist again.

Release to Google Play Store [Tuesday, 3rd week]

• Tag the latest released RC version additionally with the tag of the release (v1.0-RC2 -> v1.0) (This can be done as soon as there are no more release blockers, does not need to be on Release Day.)

• Upload signed APKs to the release page

• Create a release request in Bugzilla. You can clone this issue and need-info someone from release management.

  • Release management promotes the build from beta to the release channel. For minor releases this can happen at any time during the day. Major releases often need to be synchronized with other marketing activities (e.g. blog postings). Releases are rolled out to 99% of the population (otherwise the rollout can't be stopped).

After the release

• Check whether there are new crashes that need to be filed.
• Follow up with Product about releasing to 100%.
• In the Bugzilla bug, ask release management to set the rollout to 100%.

Room for improvement

• Automate assigning milestones to closed issues (based on date, etc) Automate some manual parts of Releng Checklist #6199
• Automate assignig eng:qa:needed to issues Automate some manual parts of Releng Checklist #6199

┆Issue is synchronized with this Jira Task

[severinrudie]

Note re: Go through the list of issues closed during this sprint and make sure they all have the correct milestone. This is every #dddd found in the git log since release 2.3.

#6354
#6269
#6390
#6468
#6284
#6087
#6466
#6441
#6462
#6414
#6450
#6441
#6443
#6369
#6376
#6405
#6097
#6309
#4126
#6359
#5917
#6210
#6153
#6341
#6268
#6342
#6230
#5959
#5958
#1487
#5710
#4456
#6003
#6344
#6348
#4775
#6311
#6343
#6294
#1298
#5896
#6272
#6160
#5128
#5678
#6249
#5848
#6047
#5733
#6188
#5104
#6229

[severinrudie]

I filtered out the #s that point to PRs. Those that point to issues are named:

• Turn on ETP for all users
• "Search with" and "Search or enter address" strings improperly positioned in RTL
• "Download" button's UI should have the Fenix specifics
• No Search Shortcuts shown if tap on URL
• Search suggestions multiply for non-default providers
• RuntimeException: Unable to create application org.mozilla.fenix.FenixApplication: java.lang.IllegalArgumentExcepti...
• Protection settings has no startPadding
• activation ping metrics are expired
• Migrate Quick Settings to LibState
• No feedback when pressing items on the copy/patse menu
• Display silent Running in Firefox notification in TWA
• Long tapping on a tab does not check it when creating a Collection
• Replace remaining android:drawableTint with appcompat version
• Onboarding remains displayed if you open tabs using the "+" button
• System Download Notification
• In-app Download Notifications
• Quick access to existing bookmarks
• total_uri_count should be added to the metrics ping
• IllegalStateException: it.campaign must not be null
• Come up with sustainable triage model & also triage
• Single tab doesn't get saved to collections
• Add Adjust campaignID to metrics ping
• Improperly displayed "No history here" after rotation
• Selected session may be rendered in ExternalAppBrowserActivity
• Black buttons in edit mode on the bookmark multi-selection toolbar
• Open from private mode notification should always open the private mode homescreen
• Improperly displayed Site permissions - "No site exceptions" after rotation
• Crash when opening new tab in CustomTab

[severinrudie]

Boiled the above down to.

- Turned on ETP for all users
- Added support for long running downloads with notifications
- Bug fixes and UI updates

🤦‍♂ so much wasted work

[severinrudie]

Hey @bifleming @liuche, the process of going through these issues took a few hours, and was very manual. I've scripted some of it, so hopefully it'll be a bit quicker next time, but it still feels like a lot given the eventual output. I'm not sure what we should be doing instead, but I think finding a better solution to this should be on our radar.

On FFTV one of our 'definition of done' reqs was adding a line to a changelog that we maintained in source control. That would definitely speed this portion up, but wouldn't give us an opportunity to clean up missed milestones / qa status on issues. Food for thought.

[severinrudie]

Per @liuche: with the beta cycle, on the 3rd Tuesday of the cycle (today) we would normally release the previous version. Since this is the first beta cycle, nothing will happen today. 3.0 will be promoted to release on the next cycle.

The release docs will soon be updated with more details regarding betas.

[liuche]

Will also need to uplift strings, like #6530

[liuche]

@baron-severin since we're planning on launching on Dec 2, I think the last string uplifts can happen this week, so all that needs to happen is tag the release and ask Relman to release.

[severinrudie]

@baron-severin since we're planning on launching on Dec 2, I think the last string uplifts can happen this week, so all that needs to happen is tag the release and ask Relman to release.

@boek and I attempted to uplift strings, but there were a very large number of merge conflicts and the vast majority of strings were for features that do not exist in 3.0. We timeboxed this to about an hour (trying several different approaches) and moved on to other work. I think 3.0 has diverged a lot more from master than is normal, given the unusual nature of this release.