Development workflow

cdhowie edited this page Jun 3, 2012 · 7 revisions
Clone this wiki locally

Development workflow

Preamble

(Or, when to branch.)

In a nutshell, create a new branch when you start working on something that does not depend on changes in any other branch. If two features can be developed separately without referencing parts of each other, separate branches are appropriate.

Permanent branches

We have three primary branches that will always exist, as they are part of the workflow:

  1. master: This is the primary development branch. Code in master should be thoroughly tested by the developer who authored it, as it serves as the obvious branch from which to create new branches.
  2. beta: This is the deployment branch for the beta site. Code in this branch is subject to being deployed to the beta site at any time. Master is the only branch that should ever be merged into beta.
  3. live: This is the deployment branch for the live site. Code in this branch is subject to being deployed to the live site at any time. Beta is the only branch that should ever be merged into live. It is expected that at least one developer (and preferably more) will have tested the code in the beta branch on the beta site before merging it into live.

There is also an svn branch that represents the latest commit in the old Subversion repository. This branch should never be touched except when importing any new commits that may be made to the old repository.

Creating a new branch

Let's say you are going to work on adding widgets to the site. You will begin by creating a new branch, based on master:

git fetch
git checkout -b myname-add-widgets origin/master

Replace "myname" with some text that represents you. This will help identify at a glance which branches are yours and which are not. It's perfectly acceptable to work with someone else on the same branch, but it's good etiquette to refrain from committing changes to someone else's branch without their permission, as that will complicate their development process when they go to push their changes. (You can always create a branch from someone else's branch if you need their changes. More on that later.)

The above command is equivalent to the more verbose:

git fetch
git branch myname-add-widgets origin/master   # Create the "myname-add-widgets" branch from master
git checkout myname-add-widgets               # Switch working directory to the new branch

Doing some work

Now you will do the work to add widgets. You may (and are encouraged) to commit periodically, when it makes sense to do so. This is not an exact science; some people will commit more frequently than others, and this is okay.

git add file1 file2                    # Stage file1 and file2 for the next commit
git commit -m 'Added some files'       # Make the commit

When you are ready to publish your changes, push them to the Github repository. If you have not yet pushed your branch, you will have to explicitly push it so that Git knows you meant to share it, otherwise it will assume you are not ready to publish it yet.

git push -u origin myname-add-widgets  # Initially pushes the branch

(The -u switch tells Git to alter your local branch to track the new remote branch. Usually this is what you want.)

Then, the next time you can just git push and Git will push all matching local branches to their respective remote branches.

Merging your branch and master

Step one: Merging master into your branch

Once you have completed your feature or fix and have ensured that it works correctly and does not cause any breakage, you are ready to merge it into the master branch. (If your code causes things to break, then other people who will create branches off of master will be starting with broken code, and that's not good. We won't get this right all the time; broken code in master is an inevitability but we should try very hard to avoid it.)

The first thing you need to do is fetch any new commits from the repository so that you are merging with the current master branch; if you merge against an older commit you will have to merge again with the latest commit in order to push.

git fetch

Now that you have the latest, merge master into your branch.

git merge master

This command may report that the branch is "already up-to-date." That means that master has not been changed since you branched it, so no merge is necessary. Skip down to the next section if this is the case.

If there are no conflicts, this will create a merge commit, which will have two parents instead of the usual one. The parents will be the most recent commit that was on your branch and the most recent on master, effectively merging their histories together. If there are conflicts because both branches changed overlapping regions in one or more files, Git will report this and will not create a new commit yet. git status will report which files have conflicts. Open each one and resolve the conflicts, then git add them to mark them as resolved. Then you will be able to git commit the resulting merge.

Note that this will not update the master branch, it will only update your branch. This is what we want. Now, you have the latest changes to master in your own branch. Re-test your code and make sure that it still works fine, especially if you had to resolve conflicts.

Once you are satisfied that the code is working, you can proceed to the next section. If more fixes are necessary, make new commits as necessary to address any problems and then continue to the next section.

Step two: Merging your branch into master

If someone else has pushed changes to master while you were doing your merge, it may be necessary to merge again. Run git fetch and make sure that master is not updated. If you see it change, repeat the previous section.

Now you are ready to merge your branch into master. This will be a so-called "fast-forward" -- if the branch you are merging in represents a commit that is a direct descendant of the branch you are merging into, Git will simply move the current branch forward to that commit.

git checkout master                    # Switch to the master branch
git merge --ff-only myname-add-widgets # Merge your branch in

The --ff-only switch will make Git abort the merge if it cannot be resolved as a fast-forward. This is just a sanity check, since Git should be able to if these directions have been followed.

Step three: Push the result

Now simply git push and your merge will be published.

If Git reports an error saying that master cannot be pushed because it cannot be resolved as a fast-forward, that means that someone else pushed some commits to master while you were preparing your merge. Go back to step one and merge them into your branch.

Detour: Branching from a branch

Sometimes it may be necessary to branch from someone else's branch, for example if your changes build on theirs. To do this, just alter the checkout command accordingly.

git checkout -b myname-add-widgets theirname-add-function-library

Now your commits will fork from their branch instead of master. Development will proceed as above, up until you want to merge your changes into master. The rule of thumb is that you should not merge your branch into master until the branch you started from is merged into master. Doing so does not pose a technical problem, since Git is built to handle complicated merge scenarios. Rather, the problem is that the author of the original branch may have a bit more work to do before their branch is ready, or may be waiting for feedback on their changes from other developers. If you merge your branch into master, you are also merging their branch into master.

If you need to branch from one of your own branches, the same rule of thumb applies: if the parent branch is not ready to be merged, neither is the child branch. However, since you are the author of both branches, you are in a better position to decide whether merging the child branch directly into master is appropriate.

Deploying to the beta site

So, you have some spiffy new changes in master and want to deploy them to the beta site so your fellow developers can test them? That's simple enough. First, do git fetch to make sure that you have the latest commits.

If you have not previously checked out the beta branch, create a local tracking branch for it.

git checkout -t origin/beta

Otherwise check out your local beta branch and make sure it's up-to-date with the remote beta branch.

git checkout beta
git merge origin/beta

Now merge master into it. This should usually be a fast-forward, but it might not be if a hotfix needed to be applied to the beta site. If a conflict happens here, something is horribly wrong.

git merge master

Push the result.

git push

Now log in to willow and become unblock. Then go into the beta deployment directory and pull in the changes.

cd utrs-beta
git pull

That's it. The beta site should now reflect what is in master.

Deploying to the live site

Deploying to the live site is almost the same, except you merge beta into live.

git fetch

If you have not checked out the live branch before, create it.

git checkout -t origin/live

Otherwise check it out and update it.

git checkout live
git merge origin/live

Do the merge and push the result.

git merge beta
git push

Update the live deployment directory on willow.

cd utrs-live
git pull

Hotfixing

Sometimes there will be an error so critical that it needs to be fixed on the live site right now and there isn't time to follow the master-to-beta-to-live process. Usually this means a security, privacy, or critical functionality problem. If so, go make the change on the live site first and worry about how to get the change into the code repository later. The repository can wait.

Ok, so the site is fixed. Whew. Send out an email to the developer mailing list and let them know what happened, and that they shouldn't touch the deployment directory until you sort out the code.

The deployment directories do not have write access to the code repository, so you cannot simply commit your changes and push them up. However, we still need to create some commits so we can ship them back to your development repository.

The easiest way to obtain the commits in your development repository is to add the deployment directory as a remote. We'll do that a bit later.

In order to make sure that the commit gets created with the correct attribution, set some environment variables so that Git knows who to put in the author and committer fields.

export GIT_AUTHOR_NAME='John Doe'
export GIT_AUTHOR_EMAIL='johndoe@example.com'
export GIT_COMMITTER_NAME="$GIT_AUTHOR_NAME"
export GIT_COMMITTER_EMAIL="$GIT_AUTHOR_EMAIL"

git add the files you fixed and git commit the result. Use git log --format=fuller if you want to verify that Git has attributed the commit correctly.

Now, go back to your development repository and add the repository on willow as a remote.

git remote add utrs-live accountname@willow.toolserver.org:/home/project/u/n/b/unblock/utrs-live

Replace accountname with your toolserver account name. Now your development repository knows about the live deployment repository, so it can pull changes directly from there. git fetch utrs-live to pull in your change from the live deployment repository. Now, we need to apply your change to master. We cannot merge live into master, since this breaks our deployment workflow. We always merge master into beta, and beta into live. We never go the other direction.

To deal with this situation, Git has the command git cherry-pick. This will take exactly one commit from somewhere else and apply it to your current branch, as though that commit were a patch. So, check out master and then cherry-pick the hotfix. It would probably be a good idea to make sure you have the most recent master first.

git fetch
git checkout master
git merge origin/master
git cherry-pick utrs-live/live
git push

If the cherry-pick results in conflicts, you will have to resolve them and then git commit the result, just like you do for conflicted merges.

Now master has the fix too, hurrah! It would probably be a good idea to merge master into beta and update the beta site, if there is any possibility that the bug could be exploited there. There is no need to merge beta into live since live already has the fix; we can wait and merge later, when we have something to promote from beta to live.

Appendix A: Useful Git commands

Git has a variety of commands that are extremely useful but not advertised very much. Here is a list of some of them, in no particular order.

  • git add -i: Starts an "interactive add" session that will allow you to interactively stage files for commit using a simple interface.
  • git add -p file1 file2: Stages parts of file1 and file2. This will run a diff between the file in the working directory and the one that is staged for commit, allowing you to select only certain hunks for staging. If you were working on two different features on one branch, wound up changing the same file for both features, and want to split the changes up into two branches, this is a great way to selectively stage only parts of a file.
  • git log --all --graph --source: Shows the commit history of every branch, along with a text-based graph indicating branch and merge points. Each commit is annotated with one branch identifier indicating which branch has that commit. (If two or more branches are currently on the same commit, only one will be shown.)
  • git show-branch -a: Shows concise history of every branch, up until the common ancestor commit of all branches. The output of this command is difficult to read at first, especially if you are not totally familiar with what the terms "commit" and "branch" mean in the context of Git.
  • git fetch --all: Asks Git to fetch from every remote repository it knows about. If you have only the default "origin" then the --all switch won't do anything different.
  • git fetch --prune: The --prune switch causes Git to delete any remote tracking branches that no longer exist remotely. For example, if someone creates a topic branch theirname-some-feature and later deletes it on Github after it is merged into master, your tracking branch origin/theirname-some-feature branch will continue to exist until you prune. You can combine the --prune and --all flags to fetch from every remote and prune all remote tracking branches at once.