Skip to content
This repository has been archived by the owner on Apr 3, 2020. It is now read-only.

Rebasing our forks

Jump to bottom
Raphael Kubo da Costa edited this page May 16, 2016 · 96 revisions

As described in our Release Methodology page, Crosswalk follows a release process similar to Chromium's. Part of the release of Crosswalk it involves rebasing our Chromium and V8 forks on top of a certain upstream release.

This article shows how perform these rebases and what pitfalls to look out for, as well as how to update Crosswalk itself once the forks have been rebased.

Before you begin

Some skills are essential in order to rebase the code without losing half of your hair or running away in despair.

First of all, you must be familiar with Chromium's release process (which is what we have based our own relase methodology on). Particularly important things to know include:

  • Chromium's git repository organization:
    • The master branch is where all development happens.
    • Development is stabilized in branches: no new features are introduced, and this is where final releases are created from. In git, the branches are in refs/branch-heads/<NUMBER>.
    • Releases (canaries, betas, stables etc) are tags in refs/tags/<RELEASE NUMBER>.
    • Branches and tags are not fetched by default, contrary to the defaults used by most git projects in the world.
    • DEPS is not updated in a branch after it is created from master. It is updated for releases, though (a tagged release simply points to a commit on top of a branch that updates those two files).
  • V8's development happens in git, with release branches accompanying each Chromium release. For example, Chromium 34 uses V8 from the 3.24 branch. In git, these branches are stored in refs/branch-heads/<NUMBER>.

You also need to be familiar with OmahaProxy. Understand which row you are supposed to choose and what the columns mean. If you are rebasing Crosswalk's dependencies for a new cycle (ie. you are working on what Crosswalk's master branch is going to be based on), you should check the linux-beta row in OmahaProxy. Likewise, if you are simply maintaining Crosswalk's stable branch and are thus updating from one Chromium release to another in the same milestone, use the linux-stable row. From there you will see which is newest chromium version in specific type of release.

Finally, spend some time getting to know git better. Do not follow the instructions below blindly. Specifically, make sure you know how references and refspecs work.

Branch names in chromium-crosswalk and v8-crosswalk

When creating new branches or updating existing ones, it is important to pay attention to Crosswalk's branch naming conventions for the forks. They apply to all our forks (chromium-crosswalk and v8-crosswalk).

The most basic distinction is between the master and the crosswalk-* branches:

  • master, as usual in a git-based workflow, is the current development branch and the one everyone works on at the moment. It corresponds to the fork versions Crosswalk's master branch is using.
  • When rebasing to track a different release, the previous branch should be backed up and preserved as crosswalk-<version>/<release_number>. For example, crosswalk-1/28.0.1500.36 corresponds to all the commits we made to chromium-crosswalk and v8-crosswalk while we were tracking Chromium's 28.0.1500.36 release.

In other words, crosswalk-<version>/<release_number> simply contains a certain number of commits made by Crosswalk contributors on top of an upstream branch.

A special note about chromium-crosswalk backports

Sometimes, we need to backport Chromium commits that are not present in the Chromium branch we are tracking (it may have been landed only in master, for example). In this case, we create a backport commit with git cherry-pick or git format-patch/git am. The actual tool you use does not matter, but in general the backport retains the same commit message, with "[Backport]" prepended to the first line. In other words, "PRESUBMIT: Determine checkperms.py's path using input_api" becomes "[Backport] PRESUBMIT: Determine checkperms.py's path using input_api".

The important thing to note is that these backport commits MUST NOT contain the "Cr-Commit-Position" line from the original commit message due to a limitation in how the lastchange.py Chromium script determines the latest upstream commit hash: if a backport contains a "Cr-Commit-Position" line, lastchange.py will think this is the latest upstream Chromium commit in the repository. This hash (which does not exist upstream) is later used by the DevTools code to retrieve HTML/CSS/JS resources for the remote debugger, which obviously fails.

Bottomline: always make sure the last commit to contain a "Cr-Commit-Position" line is actually from upstream (generally, "Publish DEPS for Chromium xyz").

Rebasing v8-crosswalk

Let's start with v8-crosswalk, which is the lowest layer of dependency we need to take care of. Most of the time, rebasing V8 should be trivial: not only are fork-specific commits discouraged in general, but V8 normally serves Crosswalk's purposes just fine without requiring any changes from our part.

  1. Fork v8-crosswalk.

    In case this was not obvious, make sure you have your own fork of the v8-crosswalk both for experimenting and also for submitting pull requests and exercising the bots. Do NOT push your changes directly to crosswalk-project/v8-crosswalk.git directly without testing and talking to people first!

    And, if you haven't done so, add your remote to your checkout:

    cd v8
git remote add my-fork git@github.com:myusername/v8-crosswalk.git

  2. Determine the new Chromium branch and revision that are going to be used.

    Assuming we are tracking linux-beta and it is now at release 40.0.2214.69: the v8 version column in OmahaProxy says the version number is 3.30.33.13, so the V8 tag we are interested in is called 3.30.33.13.

  3. Back up the existing master branch.

    As mentioned above, since the master branch is going to have its history changed, it must be backed up into a history branch first. For example, if we are currently tracking Chromium release 40.0.2214.28, a branch called crosswalk-11/40.0.2214.28 must be created. You can find the Chromium version number in src/chrome/VERSION (i.e. in chromium-crosswalk). You can check the Crosswalk version in src/xwalk/VERSION.

    Assuming we are currently tracking Chromium release 40.0.2214.28 in master:

    git fetch origin
git reset --hard origin/master  # Make sure you are at the tip of the tree.
git branch crosswalk-11/40.0.2214.28 origin/master
git push my-fork crossalk-11/40.0.2214.28

    If you track a beta/stable branch, the backup branch is already there, but you have to create a branch for the new version (let's assume the new version is 40.0.2214.69 as mentioned above):

    git fetch origin
git branch crosswalk-11/40.0.2214.69 crosswalk-11/40.0.2214.28

  4. Fetch the new V8 branch and create a new upstream branch.

    git fetch https://chromium.googlesource.com/v8/v8.git +refs/tags/3.30.33.13:upstream-copy-mine

    Verify the new branch upstream-copy-mine has been created by running git branch.

    It is always a good idea to double-check that the SHA1 hash of upstream-copy-mine matches the V8 hash specified in Chromium's DEPS file. Since we have not updated our chromium-crosswalk checkout yet, you can check the hash online in chromium.googlesource.com. For example, you can see the DEPS file for the 40.0.2214.69 release here: chromium/src/+/40.0.2214.69/DEPS. In this file, we can see the following snippet:

    # ...
'src/v8':
  (Var("git.chromium.org")) + '/v8/v8.git@aee41f5d4de09a9ffc245eaea459a75331b6f94d',
# ...

    As the same branch can be used for more than one release, the commit at the tip of the branch might not be the one corresponding to the release we want. Use git log to determine the SHA1 hash determined above, and then reset to it:

    git checkout upstream-copy-mine
git reset --hard <SHA1>

  5. Rebase existing fork-specific changes in master on top of the new upstream branch.

    1. In the trivial case, we have no fork-specific commits on top of the upstream ones (yay). This means both master (or crosswalk-11/40.0.2214.69 if you are not working on master) and crosswalk-11/40.0.2214.28 point to the same commit as upstream-copy-mine). Updating master should be very simple: just make it point to the same change as the new upstream branch. You can check that if there is crosswalk related commits at the top of git log.
    git checkout master  # or crosswalk-XY/<number>, if you are not working on master
git reset --hard upstream-copy-mine

    1. If we do have commits on top of the upstream ones, we need to check which of them are still relevant and re-apply those which are.

      This process involves some manual work. Use git log and any other tools at your disposal to study the Crosswalk-specific commits we have. In some cases, their commit messages say "this commit will not be needed once we rebase", or "this commit is only necessary until release XX.YY". Take note of them, because they will probably not be needed anymore. After that, use git rebase -i to rebase our commits on top of the new branch. It will try to add a lot of previous Chromium commits as well, so you need to remove a handful of lines (anything before the first Crosswalk-specific commit belongs to the previous Chromium branch and must be removed) and check which commits should actually be added.

      Note that there some manual effort is required here: there can be merge conflicts, so you need to check the commits, remove some and adjust some others. Also note that the automatic merge commits from GitHub will be lost.

    git checkout master  # or crosswalk-XY/<number>, if you are not working on master
git rebase -i upstream-copy-mine # Choose the right commits, resolve conflicts.

  6. Push your new branches to your fork.

    git push -f my-fork master

Rebasing chromium-crosswalk

The rebasing process for chromium-crosswalk is very similar to v8-crosswalk's. The differences are in some branch names upstream and the fact that it is much more likely that we have commits on top of the upstream ones.

The parts of the process that are similar to v8-crosswalk have shorter descriptions here. Please refer to the v8-crosswalk section for more detailed explanations of each step. It's not possible to emphasize enough how important it is not to follow the steps blindly, so read up and understand what is going on first.

  1. Fork chromium-crosswalk.

    Once again, fork the repository if you haven't done it yet, and DO NOT push your changes directly to crosswalk-project/chromium-crosswalk.git directly without testing and talking to people first!

    And, if you haven't done so, add your remote to your checkout:

    git remote add my-fork git@github.com:myusername/chromium-crosswalk.git

  2. Determine the new Chromium release that is going to be used.

    Assuming we are tracking linux-beta and it is now at release 40.0.2214.69. This is the tag/release we will rebase onto. Chromium's git repository uses tags a bit differently than most git projects: while most project use a tag to refer to a commit part of a certain branch and can be accessed by traversing this branch, Chromium's release tags point to a single commit descending from a branch but not referenced by it. The sole purpose of this commit is to update DEPS so that Chromium's dependencies are at the correct revisions.

    In other words: looking at the "2214" part of 40.0.2214.69, it is possible to see that it comes from the branch 2214. In git, it means one can fetch and follow refs/branch-heads/2214 containing all commits from master that have been merged into the 1599 branch, but with an outdated DEPS file. The tag refs/tags/40.0.2214.69, on the other hand, points to a commit fixing DEPS that is a direct child of the commit in refs/branch-heads/2214 but is not part of that branch.

  3. Back up the existing master branch.

    Assuming we are currently tracking Chromium release 40.0.2214.28:

    git fetch origin
git reset --hard origin/master
git branch crosswalk-11/40.0.2214.28 origin/master
git push my-fork crosswalk-11/40.0.2214.28

    If you track a beta/stable branch, the backup branch is already there, but you have to create a branch for the new version (let's assume the new version is 40.0.2214.69 as mentioned above):

    git fetch origin
git branch crosswalk-11/40.0.2214.69 crosswalk-11/40.0.2214.28

  4. Fetch the new Chromium tag and create a new upstream branch.

    git fetch https://chromium.googlesource.com/chromium/src.git +refs/tags/40.0.2214.69:upstream-copy-mine

    Verify the new branch upstream-copy-mine has been created by running git branch.

  5. Rebase existing fork-specific changes in master on top of the new upstream branch.

    1. In the trivial case (ie. we have no commits on top of upstream):
    git checkout master
git reset --hard upstream-copy-mine
    1. If we do have commits of our own, use git log to check if some of the commit messages say certain commits can be safely removed when moving to a newer Chromium release. For example, the first commit, which says "Publish DEPS for Chromium XXX", should always be skipped:
    git checkout master
git rebase -i upstream-copy-mine # Choose the right commits, resolve conflicts.

  6. Push your new branches to your fork.

    git push -f my-fork master

Updating Crosswalk itself

Now that the forks themselves have been updated, we need to work on the Crosswalk part of the rebase. It can be divided in two parts: first, smoke-test your fork changes by building content shell, then update Crosswalk's code since some of the Chromium (and maybe V8) features it uses have changed (this is very much likely if you are tracking a new milestone, whereas stable updated should be fairly painless in this regard).

  1. Reset your copy to the upstream one

    cd src/xwalk
git fetch origin
git reset --hard origin/<BRANCH>

  2. Update version numbers in Crosswalk.

    Quick recap: Crosswalk follows a MAJOR.MINOR.BUILD.PATCH versioning scheme. See our Release Methodology page for an explanation of each of those numbers.

    When moving to a new Chromium milestone, we need to bump the MINOR number: this is the major number of the new Chromium release we are going to track. For example, if Crosswalk's current version is 1.28.52.3 and we are now tracking Chromium 30.0.1599.66, the new Crosswalk version number should be 1.30.52.3.

    These version numbers need to be updated in the following files:

    • src/xwalk/VERSION

    For new releases in the stabilization branch where we are just moving to a more recent version of the same Chromium milestone, nothing needs to be done.

  3. Update DEPS.xwalk.

    This should not require many explanations: it is the file used to generate .gclient-xwalk and determines where we fetch Chromium and V8 from and at what revision.

    First of all, check if there are entries there that could be removed (for example, there could be an entry saying "Delete the dependency below once we track Chromium >M30"), and remove them.

    After that, chromium_crosswalk_rev and v8_crosswalk_rev need to point to the new SHA1 hashes in chromium-crosswalk and v8-crosswalk, respectively (including the Crosswalk commits that might exist on top of the upstream ones). You can use git rev-parse HEAD or git log to determine the SHA1 hashes.

    ONLY ON CROSSWALK < 16: Additionally, you have to set blink_upstream_rev, which is the SVN commit number of the latest upstream Blink commit we are pulling. It can be obtained by running git log in blink-crosswalk and checking the line that says something like git-svn-id: svn://svn.chromium.org/blink/branches/chromium/2311@191638 bbb929c8-8fbe-4397-9dbb-9b2b20218538. The 191638 part there is the SVN commit number.

  4. Smoke-test the fork updates.

    The first way to verify your rebases went well and nothing is broken is to build Chromium's content shell. You should already be familiar with content shell, so let's go directly to the commands:

    gclient sync -v
# gclient should now checkout a new 30.0.1599.66 directory and
# then fetch new versions of a lot of third-party dependencies.

cd /path/to/chromium-crosswalk
python build/gyp_chromium
ninja -C out/Release content_shell

    There is nothing too special above, you just need to tell gclient to fetch the new dependencies, build content shell and then run it to verify everything is working. The exact way you build the content_shell target (with ninja or with make etc) is irrelevant. Building in debug mode is recommended in order to build assertions and other parts of the code ignored in a release build.

After things are working, update Crosswalk's DEPS.xwalk and go back to the tests described above.

  1. Try building Crosswalk and its tests.

    Once you are sure content shell is OK, it is time to verify Crosswalk itself. Start by trying to build xwalk and its tests, then running them.

    cd /path/to/chromium-crosswalk
python xwalk/gyp_xwalk # Optional arguments etc etc.
ninja -C out/Release xwalk_builder
./out/Release/xwalk_unittest # and others

    Again, the exact command line arguments are not described above, and can be found in other pages. The important part is that you must build Crosswalk and test it. Try doing that with as many platforms as you can, such as Android and Windows as well.

    If you are working on a stable branch, the whole process should not have unexpected bumps.

    On the other hand, if this is a rebase on top of a new Chromium milestone for Crosswalk's development branch, it is very likely that Crosswalk will fail to build. This is the time to start adapting Crosswalk's source code to the changes in Chromium itself. You can also git commit your current changes, push them to a branch in your Crosswalk fork and ask for more people to chime in and help, as Chromium's code can change a lot between releases at times.

    Rinse and repeat until everything builds, all tests pass and Crosswalk seems to be working.

Push your forks

Once everything is working, you can push your chromium-crosswalk and v8-crosswalk changes to crosswalk-project if you haven't done so yet.

# Assuming origin points to git@github.com:crosswalk-project/{chromium,v8}-crosswalk.git
cd /path/to/chromium-crosswalk/v8
git push -f origin master  # Or your beta/stable branch name.

cd /path/to/chromium-crosswalk
git push -f origin master  # Or your beta/stable branch name.

This should not break anything for Crosswalk users, as the SHA1 hashes referenced in the Crosswalk repository are still present in the forks.

Send a Crosswalk pull request

By now, you should have verified that all forks are working as expected, and you can submit the final Crosswalk pull request moving Crosswalk to the new Chromium.

  1. Make sure you are on top of the tip of the tree. Your Crosswalk branch should be on top of the latest master (or crosswalk-16 etc if you are working on an update to a beta branch).

    cd /path/to/src/xwalk
git checkout next  # Or whatever your branch name is
git fetch origin
git rebase origin/master  # Or origin/crosswalk-16 etc

  2. Check your coding style.

    If you have several commits adjusting Crosswalk to the new Chromium (particularly true if you are working on a new Chromium milestone and are thus working on the next branch), you should make sure you have not introduced any coding style regressions. The easiest way to do that is to run the style checker our Trybots run, which unfortunately is located in the Buildbot code itself:

    cd /somewhere
git clone git@github.com:otcshare/build-infrastructure.git
cd /path/to/src/xwalk
git checkout next
cd /directory/above/src
# "master" below should be "crosswalk-16" or whatever the branch you want to target with
# your pull request.
python /somewhere/build-infrastructure/scripts/slave/xwalk/lint.py --repo=xwalk --base=master

  3. Create one single, final commit with everything.

    We need to squash all commits that went into Crosswalk into a single one. The reasoning behind that is that if we send a pull request with several commits adapting the code to a new Chromium release, it means that all commits before the last one will result in a broken build. This makes bisecting and reverting insanely difficult. Consequently, it is very important that you create a long, detailed commit message explaining what went into during this rebase, things to watch out for, Chromium CLs that resulted in changes. You should also thank people who may have helped you, and mention any JIRA tickets that can be closed with your commit.

    cd /path/to/src/xwalk
# Create a new branch for the squashed commit
git checkout -b squashed-next next
git rebase -i master  # Or whatever the base branch is
# Squash everything, write a nice commit message.

  4. Send the pull request, wait for the bots, merge and go celebrate!

Clone this wiki locally