Note project workflows
Clone this wiki locally
This page contains some information about the workflows used in Timelapse development, as well as related workflows for dealing with WebKit, GitHub, issues, pull requests, and patch series.
Developing WebKit with Git
The WebKit project's official repository is managed by Subversion, but many people use the git-svn extension to develop WebKit with git. So, many of the build systems and WebKit helper scripts are set up to interface with Git nicely. Here are some important tips, and then some more expansive links.
Creating and configuring a repository
To use GitHub for Timelapse, create a fork of
burg/timelapseon the Github website. (Consult GitHub help if you don't know what that means.)
If you are new to using
git, you'll need to configure your local git settings:
git config --global user.name "Foo Bar" git config --global user.email "email@example.com"
- Initialize an empty repository on your development machine (change the path below):
mkdir -p ~/repos/timelapse/ && cd ~/repos/timelapse/ git init
- Configure your local repository's remotes:
git remote add upstream git://git.webkit.org/WebKit.git git remote add github ssh://firstname.lastname@example.org/USERNAME/timelapse.git git remote rename origin github
Next, fetch the
upstreambranch containing vanilla WebKit, and the
timelapsebranch containing Timelapse:
git fetch upstream master:upstream git fetch github timelapse:timelapse git checkout timelapse git branch --set-upstream timelapse github/timelapse
Note that this does the initial pull from the WebKit git mirror, rather than GitHub. The repository is over 5GB in size, so using the faster WebKit project mirror is a necessity. Github will likely time out your connection, and is quite slow. After the initial fetch, it should be fine to fetch incremental changes from GitHub.
- Start building!
git-gccan be used to make your git repository take up less space and increase git's performance. Some git commands will run a "lite" version of
git-gcautomatically, but you may want to run it manually from time to time. Running it is as simple as:
- Git diffs of Objective-C can be improved with the following settings:
git config diff.objcpp.xfuncname "^[-+@a-zA-Z_].*$" git config diff.objcppheader.xfuncname "^[@a-zA-Z_].*$"
- Webkit's build system can be instructed to append the git branch name to every build. This is especially useful for keeping upstream (pure) WebKit built and up-to-date independently of Timelapse.
git config core.webKitBranchBuild (true|false) //the default is off
- Appending the branch name to the build directory can also be configured per branch, in case you want to share between feature branches but not upstream and feature branches.
git config branch.$branchName.webKitBranchBuild (true|false)
- Learning: Pro Git
- Learning: Git Reference
- Tips and Tricks for using Git with WebKit
- Proposal for moving WebKit development to Git
- Using GitHub to contribute to WebKit (proposal)
Using Issues, Tags, and Milestones
(The following guidelines are pinched from Rust's issue tracking conventions for GitHub.)
Bugs, features, fixmes, and other units of work live in the issue tracker.
Tag bugs liberally. The bug tracker here currently has weak search capabilities. Github staff has made a variety of comments suggesting "they're working on it", but in the meantime tags are our only hope.
Tags in the tracker are in groups:
[A-foo]tags mean that the bug is in the area of "foo", meaning that someone who wants to work on the foo module should look at it. These should be pretty specific areas, not vague. Some bugs will be tagged with multiple areas because they cut across areas.
[B-foo]tags mean that the issue only applies to the
foobranch. Such issues tend to be work items, or things that are disruptive to the trunk. This is to work around the lack of branch-specific issues in Github.
[E-foo]tags are a guess of the effort required by a bug. The meaning of the numbers is relative, and does not correspond to hours (since everyone works at a different pace).
[I-foo]tags are a subjective judgment of importance. A bug should be in only one
[I-foo]state. "Wishlist" is the least important: used for non-core features that would be nice to have, but don't need to be scheduled for any particular time.
[X-foo]tags mean that the bug is blocked in a particular state, such as "wanting clarification" or "RFC". These are effectively workflow-oriented tags, so we can see try to attack bugs that are stuck in a particular state of their life and could possibly become unstuck. Just "awaiting someone to do the work" is not a blocked state. A bug should be in zero or one
[B-foo]states, no more than one.
Add "FIXME (issue #NN): blah blah" in the source anywhere you see room for improvement, where #NN is the issue number in the tracker here. If you fix an issue on commit, remove the associated FIXMEs (grep for other occurrences) and put the exact phrase "Closes #NN" (with that capitalization) in the commit message and github will pick it up and link to the commit, close the issue.
Milestones are used to group issues by deadline or functionality goal. For example, they have been used in the past to keep track of bugs that block demos for a paper submission.
(The following is adapted from Rust's wiki page on git workflow)
In general, we prefer to use pull requests because they integrate nicely with issues and facilitate code review. Pull requests and issues can be automatically closed by GitHub when text such as "Fixes #5" is included in the merge commit.
When submitting pull requests it is important to know your branch may be rebased, which will change its history and make your branch's history invalid.
One possible workflow that has worked for developers on the project is outlined below. This assumes that you have an 'origin' remote that represents your remote github repo, and a 'master' remote that represents burg/timelapse. To create the latter, you can execute:
$ git remote add master git://github.com/burg/timelapse.git
- Whenever you start working on anything new, create a new branch derived from master's
$ git checkout timelapse -b mybranch
- While working, rebase your branch forwards regularly against the updates to
$ git checkout mybranch $ git fetch master $ git rebase master/timelapse
Sometimes there are conflicts during rebasing. If rebasing will cause some of your commits to not build or otherwise make less sense then don't do it (
git rebase --abort). Instead finish the work on the current branch and submit the pull request without merging or rebasing. Do not repeatedly merge
upstreaminto your branch. We will ask you to rewrite any pull request that contains unnecessary merge nodes.
- When done, rebase one final time to the
burg/timelapse, and push your work up to github:
$ git fetch master $ git rebase master/timelapse $ git push origin mybranch
File a pull request against
burg/timelapse. Target the pull request to the
timelapsebranch. Once your pull request is filed, you can create a new branch and do something else.
timelapseinto your local repo and verify that it contains your changes:
$ git checkout timelapse $ git pull $ git log ... <look for your changes, confirm they arrived> ...
- It is now safe to delete 'mybranch' from both local and remote repos:
$ git branch -D mybranch $ git push origin :mybranch
Developing In A Downstream Fork
The Timelapse repository is downstream from WebKit and never merges commits back directly. Some functionality may be sent back upstream, but it goes through a separate patch submission and review process that is independent of any version control system. A number of workflows exist for merging in changes from upstream to the timelapse branch, and maintaining local work in the presence of 1000's of merged upstream commits.
The GitHub project is forked from WebKit/webkit. Their
master branch is the same as Timelapse's
To pull changes from WebKit into
git fetch git://github.com/WebKit/webkit.git master:upstream or git fetch git://git.webkit.org/WebKit.git master:upstream
To find the Git hash for a specific SVN commit:
This will print the hash for Subversion revision 132649:
git rev-list --grep=".*\@132649.*" upstream
This can be used to merge to specific Subversion tags or commits visible on Trac.
To find a particular commit to merge beyond the last merge:
Supposing the last merged upstream commit is
c75d4e4e07d89, we must figure how many commits in
upstream are still unmerged to the
git checkout timelapse git log upstream HEAD...c75d4e4e07d89 --pretty=format:'' | wc -l
Then, find the n_th commit after that by subtracting _n from the number above:
git log upstream HEAD...c75d4e4e07d89 --reverse -n 3934
The first entry is the _n_th commit after c75d4e4e07d89.
Merging 1000 commits from
Find the 1000'th commit from the last merged commit (see above). Then,
git checkout -b merge-upstream timelapse git merge c75d4e4e07d89 git mergetool
mergetool command will apply a merge tool to all conflicts from the merge. Once you have fixed textual merges, compiled, and verified that Timelapse works, the merge can be committed. Then, merge the merge-upstream branch into the timelapse branch and push.
Maintaining Short-term Feature Branches
When upstream WebKit is merged into the timelapse branch, several hundred (or thousand) commits will be added to the branch. So, if you start a feature branch based on pre-merge 'timelapse' and an upstream merge happens, it is unlikely that your feature branch will still apply or build cleanly. A standard
git rebase will take a very long time since git will have to analyze and apply patches to every merged commit.
A faster way is to use cherrypicking to re-create a new post-merge feature branch. Cherrypicking essentially transplants the changes introduced by a commit onto an arbitrary branch. So it will take your old feature branch commits and dump them on the post-merge feature branch, and introduce merge conflicts if the changes don't apply (like rebase).
1. git checkout -b my-feature timelapse # create feature branch 2. <add commits to my-feature> 3. <upstream merged into timelapse> 4. git checkout -b new-my-feature timelapse # create feature branch from post-merge timelapse 5. git cherry-pick my-feature ^timelapse # transplant commits from my-feature to new-my-feature 6. <resolve merge conflicts>
You can then submit
new-my-feature as a pull request, and it should apply cleanly.
Maintaining Long-Term Feature Branches
When submitting pipelined changes to upstream, it can be good to use patch series management tools. These tools support a quilt-like development workflow that allows the base commit of the patch (and the patches themselves) to change frequently in response to code review and upstream changes. We recommend using Git Guilt for git patch series, and keep a separate repository for maintaining patch history.
- Clone guilt into a local repository and install it:
mkdir -p ~/repos && cd ~/repos git clone git://repo.or.cz/guilt.git cd guilt && sudo make install
- Pull and checkout the "integration" branch from this repository. It is the base revision to which patches are applied. (This may be out of date; in this case, you should maintain your own
git pull github integration && git checkout integration
- Initialize the guilt patch directories for this branch:
- Initialize and populate the patch repository into guilt's directories:
cd .git/patches/integration git init git remote add github ssh://email@example.com/burg/timelapse-patches.git git pull github master
When it's time to save your changes to the patches:
cd .git/patches/integration <check status, create commit message, etc> git push github master
NB. Git Guilt is a stripped-down version of quilt/Mercurial Queues. For background on how it works conceptually, try reading the Mercurial Queues chapter of the Mercurial book:
Submitting Patches Upstream
Submitting patches to a big project like WebKit can be intimidating, so here are some tried-and-true steps you can follow.
Minimize Patch Size
The longer a patch is, the less likely someone will be willing to review it. Try to split your changes into smaller pieces, with each patch accomplishing a single coherent change. Fortunately, git makes it easy to rewrite history and split commits into smaller commits. See the git book chapter on rewriting history.
Follow Style Conventions
WebKit has a coding style guide; follow it to reduce the amount of unnecessary review churn. Before you submit your changes, use the style checker by invoking
Document Changes, Not Code
WebKit's unwritten documentation policy emphasizes self-documenting code, which means that comments are permitted sparingly to explain hacks, cleverness, and unintuitive code that cannot be made simpler. Beyond style guidelines, the only constant of documentation is that you must update ChangeLogs to explain what you did in each patch. The
prepare-Changelogs script generates file- and function-level changelog entries, to which you should add additional information about your change.
WebKit's bug tracker is used to facilitate patch review. You must create one bug per patch, no exceptions. Try to CC relevant reviewers so that they are aware of the bug and any patches you wish to have reviewed. When a patch is ready for review, set the r=? and cq=? flags on the attachment.
After an attachment has existed for a while, the EWS (early warning system) bots will attempt to apply, build, and test your patch. By investigating non-green bot results, you can fix your patch so that it is compatible with other WebKit ports and platforms that you may not be able to test locally. In general, clean bot results are necessary for review to be granted.
Run Tests Locally, Add Missing Tests
This would go without saying, except that WebKit has lots of different test systems. Generally speaking:
- Run Tools/Scripts/run-api-tests (with a debug build) for changes that touch the WebKit API or WebKit2
- Run Tools/Scripts/test-webkit-scripts for changes to style checker or other scripts
- Run Tools/Scripts/run-webkit-tests for changes to WebCore and WebKit1/2
Beware that not all of these test suites may run by EWS bots, so make sure you run them yourself if they are relevant to your changes.
New tests are required if you are introducing new functionality to WebKit.
webkit-patch To Deal With Patches
The tool at
Tools/Scripts/webkit-patch (specifically the
apply-from-bug commands) are really helpful for iterating on a patch during review. I recommend you learn how to use them with git via the