Skip to content

Note project workflows

Brian Burg edited this page Dec 4, 2013 · 6 revisions

This page contains some information about the workflows used in Timelapse development, as well as related workflows for dealing with WebKit, GitHub, issues, pull requests, and patch series.

Developing WebKit with Git

The WebKit project's official repository is managed by Subversion, but many people use the git-svn extension to develop WebKit with git. So, many of the build systems and WebKit helper scripts are set up to interface with Git nicely. Here are some important tips, and then some more expansive links.

Creating and configuring a repository

  1. To use GitHub for Timelapse, create a fork of burg/timelapse on the Github website. (Consult GitHub help if you don't know what that means.)

  2. If you are new to using git, you'll need to configure your local git settings:

    git config --global user.name "Foo Bar"
    git config --global user.email "foo@email.org"
  1. Initialize an empty repository on your development machine (change the path below):
    mkdir -p ~/repos/timelapse/ && cd ~/repos/timelapse/
    git init
  1. Configure your local repository's remotes:
    git remote add upstream git://git.webkit.org/WebKit.git
    git remote add github ssh://git@github.com/USERNAME/timelapse.git
    git remote rename origin github
  1. Next, fetch the upstream branch containing vanilla WebKit, and the timelapse branch containing Timelapse:
    git fetch upstream master:upstream
    git fetch github timelapse:timelapse
    git checkout timelapse
    git branch --set-upstream timelapse github/timelapse

Note that this does the initial pull from the WebKit git mirror, rather than GitHub. The repository is over 5GB in size, so using the faster WebKit project mirror is a necessity. Github will likely time out your connection, and is quite slow. After the initial fetch, it should be fine to fetch incremental changes from GitHub.

  1. Start building!

Miscellaneous Tips

  • git-gc can be used to make your git repository take up less space and increase git's performance. Some git commands will run a "lite" version of git-gc automatically, but you may want to run it manually from time to time. Running it is as simple as:
    git gc
  • Git diffs of Objective-C can be improved with the following settings:
    git config diff.objcpp.xfuncname "^[-+@a-zA-Z_].*$"
    git config diff.objcppheader.xfuncname "^[@a-zA-Z_].*$"
  • Webkit's build system can be instructed to append the git branch name to every build. This is especially useful for keeping upstream (pure) WebKit built and up-to-date independently of Timelapse.
    git config core.webKitBranchBuild (true|false) //the default is off
  • Appending the branch name to the build directory can also be configured per branch, in case you want to share between feature branches but not upstream and feature branches.
    git config branch.$branchName.webKitBranchBuild (true|false)

Links

GitHub Workflows

Using Issues, Tags, and Milestones

(The following guidelines are pinched from Rust's issue tracking conventions for GitHub.)

Bugs, features, fixmes, and other units of work live in the issue tracker.

Tag bugs liberally. The bug tracker here currently has weak search capabilities. Github staff has made a variety of comments suggesting "they're working on it", but in the meantime tags are our only hope.

Tags in the tracker are in groups:

  • [A-foo] tags mean that the bug is in the area of "foo", meaning that someone who wants to work on the foo module should look at it. These should be pretty specific areas, not vague. Some bugs will be tagged with multiple areas because they cut across areas.
  • [B-foo] tags mean that the issue only applies to the foo branch. Such issues tend to be work items, or things that are disruptive to the trunk. This is to work around the lack of branch-specific issues in Github.
  • [E-foo] tags are a guess of the effort required by a bug. The meaning of the numbers is relative, and does not correspond to hours (since everyone works at a different pace).
  • [I-foo] tags are a subjective judgment of importance. A bug should be in only one [I-foo] state. "Wishlist" is the least important: used for non-core features that would be nice to have, but don't need to be scheduled for any particular time.
  • [X-foo] tags mean that the bug is blocked in a particular state, such as "wanting clarification" or "RFC". These are effectively workflow-oriented tags, so we can see try to attack bugs that are stuck in a particular state of their life and could possibly become unstuck. Just "awaiting someone to do the work" is not a blocked state. A bug should be in zero or one [B-foo] states, no more than one.

Add "FIXME (issue #NN): blah blah" in the source anywhere you see room for improvement, where #NN is the issue number in the tracker here. If you fix an issue on commit, remove the associated FIXMEs (grep for other occurrences) and put the exact phrase "Closes #NN" (with that capitalization) in the commit message and github will pick it up and link to the commit, close the issue.

Milestones are used to group issues by deadline or functionality goal. For example, they have been used in the past to keep track of bugs that block demos for a paper submission.

Pull requests

(The following is adapted from Rust's wiki page on git workflow)

In general, we prefer to use pull requests because they integrate nicely with issues and facilitate code review. Pull requests and issues can be automatically closed by GitHub when text such as "Fixes #5" is included in the merge commit.

When submitting pull requests it is important to know your branch may be rebased, which will change its history and make your branch's history invalid.

One possible workflow that has worked for developers on the project is outlined below. This assumes that you have an 'origin' remote that represents your remote github repo, and a 'master' remote that represents burg/timelapse. To create the latter, you can execute:

$ git remote add master git://github.com/burg/timelapse.git
  1. Whenever you start working on anything new, create a new branch derived from master's timelapse branch:
$ git checkout timelapse -b mybranch
  1. While working, rebase your branch forwards regularly against the updates to timelapse:
$ git checkout mybranch
$ git fetch master
$ git rebase master/timelapse

Sometimes there are conflicts during rebasing. If rebasing will cause some of your commits to not build or otherwise make less sense then don't do it (git rebase --abort). Instead finish the work on the current branch and submit the pull request without merging or rebasing. Do not repeatedly merge timelapse or upstream into your branch. We will ask you to rewrite any pull request that contains unnecessary merge nodes.

  1. When done, rebase one final time to the timelapse branch of burg/timelapse, and push your work up to github:
$ git fetch master
$ git rebase master/timelapse
$ git push origin mybranch
  1. File a pull request against burg/timelapse. Target the pull request to the timelapse branch. Once your pull request is filed, you can create a new branch and do something else.

  2. Pull timelapse into your local repo and verify that it contains your changes:

$ git checkout timelapse
$ git pull
$ git log
  ... <look for your changes, confirm they arrived> ...
  1. It is now safe to delete 'mybranch' from both local and remote repos:
$ git branch -D mybranch
$ git push origin :mybranch

Developing In A Downstream Fork

The Timelapse repository is downstream from WebKit and never merges commits back directly. Some functionality may be sent back upstream, but it goes through a separate patch submission and review process that is independent of any version control system. A number of workflows exist for merging in changes from upstream to the timelapse branch, and maintaining local work in the presence of 1000's of merged upstream commits.

Merging upstream into timelapse

The GitHub project is forked from WebKit/webkit. Their master branch is the same as Timelapse's upstream branch.

To pull changes from WebKit into upstream:

git fetch git://github.com/WebKit/webkit.git master:upstream

or

git fetch git://git.webkit.org/WebKit.git master:upstream

To find the Git hash for a specific SVN commit:

This will print the hash for Subversion revision 132649:

git rev-list --grep=".*\@132649.*" upstream

This can be used to merge to specific Subversion tags or commits visible on Trac.

To find a particular commit to merge beyond the last merge:

Supposing the last merged upstream commit is c75d4e4e07d89, we must figure how many commits in upstream are still unmerged to the timelapse branch:

git checkout timelapse
git log upstream HEAD...c75d4e4e07d89 --pretty=format:'' | wc -l

Then, find the n_th commit after that by subtracting _n from the number above:

git log upstream HEAD...c75d4e4e07d89 --reverse -n 3934

The first entry is the _n_th commit after c75d4e4e07d89.

Merging 1000 commits from branch:upstream to branch:timelapse:

Find the 1000'th commit from the last merged commit (see above). Then,

git checkout -b merge-upstream timelapse
git merge c75d4e4e07d89
git mergetool

The mergetool command will apply a merge tool to all conflicts from the merge. Once you have fixed textual merges, compiled, and verified that Timelapse works, the merge can be committed. Then, merge the merge-upstream branch into the timelapse branch and push.

Maintaining Short-term Feature Branches

When upstream WebKit is merged into the timelapse branch, several hundred (or thousand) commits will be added to the branch. So, if you start a feature branch based on pre-merge 'timelapse' and an upstream merge happens, it is unlikely that your feature branch will still apply or build cleanly. A standard git rebase will take a very long time since git will have to analyze and apply patches to every merged commit.

A faster way is to use cherrypicking to re-create a new post-merge feature branch. Cherrypicking essentially transplants the changes introduced by a commit onto an arbitrary branch. So it will take your old feature branch commits and dump them on the post-merge feature branch, and introduce merge conflicts if the changes don't apply (like rebase).

Example:

1. git checkout -b my-feature timelapse        # create feature branch
2. <add commits to my-feature>
3. <upstream merged into timelapse>
4. git checkout -b new-my-feature timelapse    # create feature branch from post-merge timelapse
5. git cherry-pick my-feature ^timelapse       # transplant commits from my-feature to new-my-feature
6. <resolve merge conflicts>

You can then submit new-my-feature as a pull request, and it should apply cleanly.

Maintaining Long-Term Feature Branches

When submitting pipelined changes to upstream, it can be good to use patch series management tools. These tools support a quilt-like development workflow that allows the base commit of the patch (and the patches themselves) to change frequently in response to code review and upstream changes. We recommend using Git Guilt for git patch series, and keep a separate repository for maintaining patch history.

  1. Clone guilt into a local repository and install it:
    mkdir -p ~/repos && cd ~/repos
    git clone git://repo.or.cz/guilt.git
    cd guilt && sudo make install
  1. Pull and checkout the "integration" branch from this repository. It is the base revision to which patches are applied. (This may be out of date; in this case, you should maintain your own integration branch)
    git pull github integration && git checkout integration
  1. Initialize the guilt patch directories for this branch:
    guilt init
  1. Initialize and populate the patch repository into guilt's directories:
    cd .git/patches/integration
    git init
    git remote add github ssh://git@github.com/burg/timelapse-patches.git
    git pull github master

When it's time to save your changes to the patches:

    cd .git/patches/integration
    <check status, create commit message, etc>
    git push github master

NB. Git Guilt is a stripped-down version of quilt/Mercurial Queues. For background on how it works conceptually, try reading the Mercurial Queues chapter of the Mercurial book:

[http://hgbook.red-bean.com/read/managing-change-with-mercurial-queues.html]

Submitting Patches Upstream

Submitting patches to a big project like WebKit can be intimidating, so here are some tried-and-true steps you can follow.

Minimize Patch Size

The longer a patch is, the less likely someone will be willing to review it. Try to split your changes into smaller pieces, with each patch accomplishing a single coherent change. Fortunately, git makes it easy to rewrite history and split commits into smaller commits. See the git book chapter on rewriting history.

Follow Style Conventions

WebKit has a coding style guide; follow it to reduce the amount of unnecessary review churn. Before you submit your changes, use the style checker by invoking Tools/Scripts/check-webkit-style.

Document Changes, Not Code

WebKit's unwritten documentation policy emphasizes self-documenting code, which means that comments are permitted sparingly to explain hacks, cleverness, and unintuitive code that cannot be made simpler. Beyond style guidelines, the only constant of documentation is that you must update ChangeLogs to explain what you did in each patch. The prepare-Changelogs script generates file- and function-level changelog entries, to which you should add additional information about your change.

Use Bugzilla

WebKit's bug tracker is used to facilitate patch review. You must create one bug per patch, no exceptions. Try to CC relevant reviewers so that they are aware of the bug and any patches you wish to have reviewed. When a patch is ready for review, set the r=? and cq=? flags on the attachment.

After an attachment has existed for a while, the EWS (early warning system) bots will attempt to apply, build, and test your patch. By investigating non-green bot results, you can fix your patch so that it is compatible with other WebKit ports and platforms that you may not be able to test locally. In general, clean bot results are necessary for review to be granted.

Run Tests Locally, Add Missing Tests

This would go without saying, except that WebKit has lots of different test systems. Generally speaking:

  • Run Tools/Scripts/run-api-tests (with a debug build) for changes that touch the WebKit API or WebKit2
  • Run Tools/Scripts/test-webkit-scripts for changes to style checker or other scripts
  • Run Tools/Scripts/run-webkit-tests for changes to WebCore and WebKit1/2
  • Run Tools/Scripts/run-javascriptcore-tests for changes to JSC.

Beware that not all of these test suites may run by EWS bots, so make sure you run them yourself if they are relevant to your changes.

New tests are required if you are introducing new functionality to WebKit.

Use webkit-patch To Deal With Patches

The tool at Tools/Scripts/webkit-patch (specifically the upload and apply-from-bug commands) are really helpful for iterating on a patch during review. I recommend you learn how to use them with git via the -g and --git-index options.

Something went wrong with that request. Please try again.