This tool was initially written to make our branch landing strategy as painless as possible. It started as a bash script, then progressed into a very basic python script. Then it turned into a much better python script during an extensive rewrite. Then it was basically rewritten again and unit tests were added to make sure that other people could use and contribute to this tool.
A lot of design decisions in this script were fine when it was a simple script but are now a bit of a pain to work with. I intend to fix some of these issues as time permits, but patches are always welcome. Please remember when looking at this code that it was written to make incremental improvements to a previously mind-numbing manual task while focusing on other projects.
The basic flow of this program is:
- gather the requirements for an uplift (see
gaia_uplift.uplift
) - find the commits that are needed for each bug needing uplift (see
gaia_uplift.find_commits
) - determine which branches each commit is needed on (see
gaia_uplift.branch_logic
) - cherry-pick commits onto the appropriate branches (see
gaia_uplift.uplift
) - push new cherry-picked commits to the remote (see
gaia_uplift.uplift
) - comment on and set flags for each bug that has an uplift (see
gaia_uplift.reporting
)
There is also an alternate flow for branches which use merges instead of cherry-picks.
- (optionally) attempt a merge
- look at the list of new commits on the branch and try to find bug numbers
- comment on the bugs with commits and set flags
The requirements are saved into a JSON
Like many tools (git, gcc, etc) this program is implemented as a collection of subcommands to the
uplift driver program called uplift
. The important subcommands are:
show
: this command takes no arguments. It does the requirement gathering stage of an upliftuplift
: this command takes no arguments. See the section called 'finding commits' for detailscomments [uplift_report_file.json]
: this command uses the data inuplift_report.json
to replay the commenting. Optionally, pass in an alternate Uplift Report json file to work from and flag setting. Useful if there is a bug in the commenting code and you need to retry just the commentsupdate
: use the uplift program's logic to recreate a clean slate of Gaia using the cached Gaiamerge to-branch from-branch
: checkout out to-branch and merge in from-branch. This will automatically comment on the bug numbers found in the range of commits pushed into to-branchmerge-comments to-branch commit-range
: like the comments command above, just comment on bugs. The commit-range argument is the thing that the git push command outputs in the form aaaaaaaa..bbbbbbbb
This is the reason that we need a human in this process. There is currently no standardized system in bugzilla for storing the commit that fixed the bug. The known methods include:
- link to the
github.com/mozilla-b2g/gaia.git
commit - a string like "master: abcd123" which is implied to be a master commit on
mozilla-b2g/gaia.git
- similar to the above, but referencing the patch author's person remote instead of mozilla-b2g's
- link to a merged pull request as a bug comment
- link to a pull request that's been merged as an attachment
- html attachment that does a redirect to a github pull request
- a response to a complicated set of email/irc exchanges trying to coax the information from the lander
I have written a basic heuristic to find commits, but it's by no means robust. Please consider the guessing system as an aid to reduce the amount of copy and paste mistakes.
Once the program has the list of bugs that you need to figure out, you'll be presented with a list of commands, guesses and an input field. The commands should be explained with up-to-date information in the program, but in general you'll experience:
- a browser window will open for the bug
- you'll inspect the guesses (if any) and make sure that the guessed commit is actually what should be uplifted. Do a sanity check on the bug. The uplift program will make sure that the commit entered is on a valid branch and is a valid commit.
- either accept a guess or copy and paste a commit id from the browser to the program. Links to github.com commits also work (makes it easy to copy link location, or drag a link over)
- if the bug is showing up in the uplift requirements but is clearly not appropriate for gaia uplift, you can skip it permanently with the skip command
- when you've found all needed commits, type done. If there are no appropriate commits, just type done to say that you didn't find any commits
The program will move onto the next bug.
Problem: The program dies midway through finding commits
Solution: re-run uplift uplift
and when asked to reuse, add or delete select the add option
Problem: The program dies after finding commits but before pushing the new commits
Solution: re-run uplift uplift
and when asked to reuse, add or delete select the reuse option
Problem: The program dies during a push and nothing was pushed
Solution: re-run uplift uplift
and when asked to reuse, add or delete select the add option
Problem: The program dies after a push but the push was successful
Solution: Use uplift comments
to redo comments. Make sure you don't redo the uplift
Problem: my push succeeded but commenting died and I re-did the uplift and lost my commits for commenting
Solution: There is a filed called uplift_outcome_datestamp.json
that stores
the same contents as the uplift_report.json file that the comments
command uses. Copy
the correct uplift_outcome_datestamp.json
to be uplift_report.json
You are modifying one of Github's most active repositories. This repository and the branches that you are working on are what gets put on shipping phones. You are needed in this process because we don't have a standardised method to store which commits should be uplifted. If you have experience doing uplifts for Gecko, please use the same discretion you use for it.
The biggest specific warning that I have is that once you say "yes, push these
commits" you should not rerun the uplift process until all the bugs in
uplift_report.json
have been commented on and their flags set. If
you do try to rerun the uplift, you're going to find that none of the uplifts
work properly and the bugzilla comments will incorrectly say that the commit
was already on the branch.
Currently, this tool does not support installation through pip/pypi. This is a known defect and may get resolved some time. The way that this tool is used by the author is to create a virtualenv and run the tool in that environment.
/**
* Create the uplift environment.
*/
git clone http://github.com/jhford/uplift.git uplift && cd uplift
virtualenv .
python setup.py develop
If you are getting import errors, try manually installing the runtime dependencies.
cd uplift
source bin/activate
pip install pip install isodate requests PrettyTable
/**
* Initialize the uplift environment.
*/
source bin/activate
uplift uplift
I'm more than happy to review and land potential fixes.
These things make your submission more likely to be accepted:
- complete tests.
- small, logical patches
- using good libraries where necessary
- using python3 compatible libraries
- avoiding modules with C dependencies
- pep8 compliance
- underscore method naming
These things make your submission less likely to be accepted:
- major rewrites which aren't thoroughly tested
- large patches
- code that doesn't appear visually similar to existing code
- making code messier than it already is
Issues will be tracked in the github issue tracker
You should always run unit tests with make test
. Only hosers don't run unit tests
This is a sample run of a really simple uplift
$ uplift uplift Found existing requirements. Should they be used? [Y/n]: n Running Bugzilla searches ...... Finished running searches Fetching bug data + + Finished fetching bug data ================================================================================ Bug 1 of 2 ================================================================================ Needed on: v1.3 Fixed on: Type one of * guess-all: add all guesses listed below * guess-1: 1d81c792f93f886f5ea227ee1e5c44ffcb996f70 BECAUSE: crdlc@tid.es made a comment which matched the pattern /commit/(?P[a-fA-F0-9]{7,40}): ---------------------------------------------------------------------- ---------- Merged in master: https://github.com/crdlc/gaia/commit/1d81c792f93f886f5ea227ee1e5c44ffc b996f70 * sha1 commit: add a raw commit number * skip: add a bug to the list of permanently skipped bugs * delete-all: remove all commits from this bug * delete-N: delete entered commit * browser: (re)open the bug in a browser -------------------------------------------------------------------------------- Bug 973853's 0 commits: * none Enter command: 70c72c5 -------------------------------------------------------------------------------- Bug 973853's 1 commits: * 0) 70c72c50e9cc17503ece37c990a3d0242664b9b1 Enter command: done ================================================================================ Bug 2 of 2 ================================================================================ Needed on: v1.3 Fixed on: Type one of * guess-all: add all guesses listed below * guess-1: f03f667e1d92595df8bfddd8b0f0f7ec64d28230 BECAUSE: amirn@everything.me made a comment which matched the pattern /commit/(?P[a-fA-F0-9]{7,40}): ---------------------------------------------------------------------- ---------- landed in master: https://github.com/mozilla- b2g/gaia/commit/f03f667e1d92595df8bfddd8b0f0f7ec64d28230 * sha1 commit: add a raw commit number * skip: add a bug to the list of permanently skipped bugs * delete-all: remove all commits from this bug * delete-N: delete entered commit * browser: (re)open the bug in a browser -------------------------------------------------------------------------------- Bug 975917's 0 commits: * none Enter command: f03f667 -------------------------------------------------------------------------------- Bug 975917's 1 commits: * 0) f03f667e1d92595df8bfddd8b0f0f7ec64d28230 Enter command: done Uplift requirements: +--------+-------------+-------------+-------------------------------------------------------------------------------------+ | Bug | v1.3 status | v1.2 status | Summary | +--------+-------------+-------------+-------------------------------------------------------------------------------------+ | 973853 | needed | --- | Images are not displayed the second time in link and import views | | 975917 | needed | --- | [Everything.me] Tapping homebutton while in Collection Rename mode may cause freeze | +--------+-------------+-------------+-------------------------------------------------------------------------------------+ Updating Gaia Fetching updates to Gaia cache directory Deleting Gaia scratch directory Cloning Gaia scratch from cache Fetching remote references Created Gaia in 45.98 seconds ================================================================================ Attempting to uplift 70c72c50e9cc17503ece37c990a3d0242664b9b1 commit to v1.3 Sucess on v1.3 Failure on ================================================================================ Attempting to uplift f03f667e1d92595df8bfddd8b0f0f7ec64d28230 commit to v1.3 Sucess on v1.3 Failure on +--------+---------------+-------------+-------------+-------------------------------------------------------------------------------------+ | Bug | master commit | v1.3 commit | v1.2 commit | Summary | +--------+---------------+-------------+-------------+-------------------------------------------------------------------------------------+ | 973853 | 70c72c5 | 8fce95f | --- | Images are not displayed the second time in link and import views | | 975917 | f03f667 | b04d2a8 | --- | [Everything.me] Tapping homebutton while in Collection Rename mode may cause freeze | +--------+---------------+-------------+-------------+-------------------------------------------------------------------------------------+ If you push, you'd be pushing: * v1.3: 55faaae..b04d2a8 Do you wish to push? [Y/n]: y Push attempt 1 worked Commenting on bug 973853 Commenting on bug 975917