git workflow

back to home

Git Workflow used for CFS development

Overview

• Getting started
• Repository Management, Branches and Merge-Requests
• Examples
• FAQ
• Useful commands

Getting started

To get started set up your username and email, such that gitlab can correctly identify your commits

git config --global user.name "<your name (can have spaces)>"
git config --global user.email "<your email>"

A project can be cloned eiter via HTTPS or via SSH, where SSH is preferred. To clone via SSH upload your public key to gitlab in your user profile once (for details see the gitlab docs). For HTTPS you will need to enter username and password every time. The clone URL is available at every project page, e.g. for CFS and can be switched between https and ssh. To clone, open a terminal and use, e.g.

git clone --recurse-submodules URL TARGET_DIRECTORY

where TARGET_DIRECTORY is optional. You now have a local copy of the repository in the directory TARGET_DIRECTORY or CFS if TARGET_DIRECTORY was not specified.

git clone --recurse-submodules ssh://git@gitlab.com/openCFS/CFS.git
git clone --recurse-submodules https://github.com/openCFS/cfs.git

If you only want to clone a single branch (not recommended), use

git clone --recurse-submodules -b BRANCH_NAME --single-branch URL TARGET_DIRECTORY

Now cd to your repository and look around:

• git status to show the state of the working copy (clean/dirty) and which branch you are on
• git diff shows differences
• git log shows the history
• git remote -v shows all remotes (servers) you are interacting with
• git branch -a shows all branches

Also check the state of the Testsuite submodule by cd Testsuite and subsequent git status. You most likely have a detached HEAD which you should fix.

You should now be ready to check out your topic branch via

git checkout BRANCH_NAME # checkout already existing topic branch

or

git checkout -b BRANCH_NAME # create new topic branch

and start coding. Remember to use git diff and git status to stay oriented, and most importantly commit often.

The typical workflow with git is the following

1. go to your branch and get the up to date working copy (git pull)
2. change stuff, test ...
3. add files to prepare a commit (git add ...)
4. commit the files to the local working copy (git commit ...)
5. share your work, i.e. push to the remote repository (git push ...)

If you have made commits to your local repository, and there are modifications upstream (on the remote server), git has two options to combine both versions when you do a git pull. The default is to merge the branches, creating an unnecessary merge commit. Therefore, always pull using the --rebase option. This will put your local commits after the upstream changes. You can set this as the default by

git config [--global] pull.rebase true

Repository Management, Branches and Merge-Requests

We use a centralized workflow with the main long running branch master. To make sure nothing accidentally ends up on the master branch it is protected, thus only pepole with sufficient permissions can push/merge into it. Everything that goes into the master branch must pass all required tests defined in the pipeline.

Always base new work on top of master, and keep it updated with changes by rebase. All other branches are subject to change, do not base work on top of them!

To keep the repository tidy please

• choose descriptive branch names, e.g. kroppert-harmonic-balance-fem or issue-NN-sequence-step-bugfix
• assign the same branch names in the Testsuite repository (if you create tests)
• delete branches that are not needed any more

In order to get your work into the master branch, you need to

1. make sure it merges to the master branch in a fast-forward merge
2. make sure it passes the pipeline
3. create a merge request from your branch to master and assign it to a maintainer for review

We have the testing branch, which usually (if nobody is currently testing) should match the master branch. Every developer can use it to test, e.g. if a merge works without conflict. When you push to testing the pipeline will automatically run using the available build cache for the cfsdeps (faster than a complete rebuild happening for a new branch name).

Small commits to long running branches

If your fix is small and you can/want to do it in a single commit the procedure is

1. ensure that your working copy (in this case of master) is up-to-date
2. do your fix, commit it locally, test locally
3. push it to the testing branch, and create a merge request (testing->master)

# assuming you start with the master branch checked out
git status
git pull
# do your fix
git diff # see what you changed
git add
git commit # write a message
# test before pushing to the remote
git push origin master:testing

If you get an error message, that testing is not available, it might be in use by someone else. Don't force the push (-f)

Intermediate size changes

Intermediate size changes typically will result in 1-3 commits and take a few hours-days of coding (and testing). They will often implement a particular new feature (one commit), and fix something related on the way (another commit). The result will be merged in a single (fast-forward) merge (via a merge request).

Such changes which will require some work locally (e.g. fix a small bug) one will typically create a local branch:

git checkout -b bug_123

Now you can try out stuff, commit locally whenever you want, and test the implementation. You can do this by staging your changes and committing them via

git add FILE # stage specific file
git commit -m "Descriptive but short commit message"

Or you can do both in one step (if you want to commit everything that has changed since your last commit):

git commit -a -m "fixed one thing, broke some testcases"
git commit -a -m "this works with test XXX"
git commit -a -m "all tests pass"

Avoid committing with -a (or git add .), since this often results in un-realted stuff accidentally bundled into a single commit!

Once you are satisfied with the implementation, it's time to share it. You can do this by pushing your branch to the server (the git remote called origin)

git push [origin [bug_123]]

where a new branch will be created.

In order to get your changes into the master branch they should be reorganized into a tidy history, which can be merged fast-forward on top of the master branch. Before we start reorganizing our branch we should make a backup by creating a backup-branch based on the one we want to backup by

git checkout bug_123
git checkout -b bug_123_backup

Next, we will use the interactive rebase functionality to do reorganize the history (see the rebase example). First make sure your local master branch is up-to-date (git fetch or checkout master & pull), then

git checkout bug_123

After checking out the branch take a look at the history we want to rewrite via

git log

In this case, we just want to rebase our work on the master (or origin/master) via

git rebase -i [origin/]master

Now your editor opens and you can organize your commits as shown in the rebase example. Typically, we want to squash some commits and change the commit message, hence, in the editor that opens we replace pick with s or squash to combine commits. Git will apply the first commit (first line) and then work its way downwards. If we want to combine e.g. all commits into one, we leave the first pick and replace every action with s

pick abcd12345 fixed one thing, broke some testcases
s dcba12345 this works with test XXX
s abcd54321 all tests pass

By proceeding we can also change the commit messages. Git will include all old commit messages but we can just replace them with a more descriptive one for the new (larger) commit. Afterwards, we can verify that our rebase worked by looking at the history again

git log

Finally, we can test our implementation. Before using the pipeline one should test locally with the help of Ctest. The next step is to trigger our CI/CD pipeline on the server to ensure all compiler versions and build configutations work.

Create a merge request (for bug_123 into master) to start the pipeline tests and get into the review process. Since the pipeline should run for each (fast-forward) commit of your merge request, you should push the commits in sequence:

git push first-commit-SHA:refs/heads/bug_123
git push ... # until all commits arrived

In the case that you already pushed something to the remote and have to run the pipeline for a certain commit (e.g. when you have to change something based on the reviewer's comments), you can use the force-push option - see CI/CD pipeline for a detailed description.

After your changes are merged, delete all associated branches you have created in the process (e.g. branch bug_123).

Short instruction sequence that proved to be useful: https://tech.serhatteker.com/post/2019-06/rebase-master/

Long running feature branches

If you plan on implementing a more complex feature (e.g. new PDE), please discuss these changes prior to your start in an issue. This gives other developers the possibility to comment on the proposed implementation and to prevent two people working on the same thing. Otherwise long running feature branches the basic concepts of the previous chapter intermediate size changes still apply. The main difference is that now we are in a situation where the new implementation will take more time, and consist of 3 or more commits. Typically, your feature branch will be merged into master by a merge commit (not fast-forward).

Often, the master branch will evolve during the feature development time. Therefore, it will be important to keep watching out for possible conflicts. This can be done in various ways, e.g.

• periodically do a test merge with the current master locally
• periodically rebase your work on top of the master branch

Long running feature branches are typically not necessary.

Working with Submodules

A git-submodule is a reference to another git repository within the main repository. Git only stores the SHA and the URL of the submodule-repo. We use this for the Testsuite.

To update a git-submodule from the main repo run

git submodule update

To check the state of the Testsuite submodule from the main repo run

git diff -- Testsuite

which will show you if the commit checked out in the Testsuite repo (=submodule) is what is referenced in the CFS repo of if there are changes. For more detail you can show differences in the submodule by (working for git >= 2.11]

git diff --submodule=diff

One way to get the Testsuite submodule into the correct/wanted state is to treat it like a stasnard git repo. You can simply cd into the submodule folder and work with it like an usual git working copy:

cd Testsuite
(git checkout master)
git pull
cd ..

Typical errors with the Testsuite submodule are:

• commit of Testsuite changes although you did not change anything, typically to an old Testsuite SHA. This is caused by git not updating submodules in the default behaviour: Always check what is staged for commit!
• merge conflicts in the Testsuite caused by changes both in master and feature branch: Make sure your Testsuite changes are on top of master (in the Testsuite repo), then commit the correct SHA (of your Testsuite version) and add it in the CFS repo to resolve the conflict.
• detached HEAD in the Testsuite submodule

A frequent source of trouble is a git setting to update the submodule automatically on git pull in the main repo, set e.g. globally by

git config --global submodule.recurse true

this usually causees problems on non-clean repos, so be careful (git refuses to pull CFS because of changes in Testsuite). Therefore, use this setting with caution!

Examples

Creating a new testcase

1. go to the testsuite (submodule), create the testcase, when you are ready to commit
2. make a new branch git checkout -b my-great-test
3. add files, commit (git add ..., git commit)
4. test locally and push when ready
5. go (up) to the main cfs repo and create a matching branch (cd ..; git checkout -b my-great-test) or checkout your feature branch
6. commit the changes in the Testsuite submodule: git add Testsuite; git commit
7. push
8. to bring stuff to master, make merge requests for both the testsuite and the cfs master, e.g. via the web frontend. Do NOT click 'Squash commits' in the web frontend for the test suite.

In case master changed during working on the testsuite branch point 4 (push when ready) is more specifically

git rebase -i origin/master
git push --force-with-lease origin +my-great-test

rebasing

Say you have created a few commits on your branch bug_123 and want to clean up the history: you need to rebase

1. find out on top of which commit your changes should go (e.g. git log or some git gui), note down the ref (SHA, branch, or tag) to rebase on. Attention: to rebase on top of the most resent master you need to pull master first!
2. check out your branch git checkout bug_123 and start an interactive rebase by git rebase -i <ref-to-rebase-on>
3. Now you will see all commits you have done, and you have to choose what to do with them. Typically you will use some of the following options:

• squash to combine a commit it with previous ones, or
• amend to modify the commit and/or its commit message, or
• drop to get rid of the commit alltogether (just delete the line), or
• you can re-order the commits (by changing the line order).
• If you want to end with a single commit, simply select squash for all but the fist one. Finally, save the file and git will start the rebase.

4. Whenever git stops, e.g. for amend, follow the instructions in git status until the rebase is done.

For details on rebasing see Atlassian rebase tutorial.

Squash intermediate branch to a single commit

In case your intermediate branch consists of several commits (backup, fixing, ...) and you want them to result in a single commit on the master, it is more convenient to merge instead of rebasing and squashing everything but the first. In case there is a corresponding change to the testsuite rebase, push and create a MR first. We assume feature_123 is commited.

1. git checkout master
2. git merge --squash feature_123
3. In case there is a corresponding teststuite: git add Testsuite
4. git commit
5. git push origin master:feature_123 --force-with-lease or git push origin master:testing --force-with-lease
6. create merge request

If you fear step 5., a similar approach is to create a third branch, which takes the squashed information within a single commit:

1. git checkout feature_123
2. git log
3. git checkout master
4. git checkout -b feature_123_squash
5. git merge --squash feature_123
6. git push --set-upstream origin feature_123_squash
7. create merge request

amend - changing the last commit

Say you noticed a typo in you last commit message, or you forgot a file

1. make the necessary changes, e.g. git add my-forgotten-file
2. commit them with git commit --amend

correcting single commits: fixup & autosquash

Say you submitted a merge request consisting of several commits, but a reviewer notices some silly typos that should be fixed. You could do an interactive rebase, but the fixup & autosquash feature is probably more convenient. It works like this

1. check which commit needs fixing and note down the SHA, e.g. git log
2. do the fix and commit it as a fixup commit, e.g. git add my-modified-file, git commit --fixup SHA
3. now do a autosquash rebase on top of the commit before the one you wanted to fix (typically master is suitable), e.g. git rebase --autosquash --interactive master

For details see https://blog.sebastian-daschner.com/entries/git-commit-fixup-autosquash.

integrating a long running branch via a merge commit

Suppose you have worked on your branch for a long time, and your history already is tidy and ready to be integrated into the master branch. It looks somewhat like the following

A -- B -- E -- F        master
      \
       C -- D -- G      feature

where you branched of from master at B, and added C-D-G on your feature branch. Master has evolved in the meantime, commits E-F were added.

Your commits are small, meaningful, and instructive adding valuable information to the history. In this case you can create a merge commit, and push it to the merge request for testing & integration to the remote master branch.

1. check out the master branch git checkout master, git pull to ensure it is up-to-date
2. merge your feature branch locally git merge feature, resolve possible conflicts in the process
3. push the result to the remote feature branch: git push origin master:feature
4. open a merge request ...

The result is the following situation

A -- B -- E -- F -- M   master
      \           /
       C -- D -- G      feature

where the merge commit M is the tip of your local master branch. The master branch on the remote (origin/master) is still at F until the merge request is accepted.

In case the pipeline fails or there are some required changes in the review, you need to

• undo the merge locally: git reset --merge ORIG_HEAD
• incorporate the changes with fixup commits
• re-do the merge locally
• (force) push to the merge request branch

Moving branches from cfs-dev to the gitlab openCFS repository

The following description (based on Andreas Wurzinger) moves local repository. It is meant for branches started on cfs-dev which shall be continued on gitlab openCFS.

1. make sure to have all your branches (that you want to keep) locally available (for both CFS and Testsuite repos)

# Show all local branches
git config -e

1. download missing branches by checking them out

git checkout <branch name>

1. change the remote url of your local git:

cd <CFS_dir> # (eg. ~/CFS_SRC/CFS)
git remote set-url origin git@gitlab.com:openCFS/cfs.git
# Check
git remote -v
# Do the same for Testsuite
cd Testsuite
git remote set-url origin git@gitlab.com:openCFS/Testsuite.git
# Check
git remote -v

1. push local branches to the new repository (for both CFS and Testsuite repos)

git push –u origin <branch name>

Working with multiple remotes

When you clone a git repository, a single default remote called origin is configured. It refers to the URL (or location) you cloned from. When you want to interact with several remote repositories (servers) you need to add additional remotes. This is a common case when you cloned from your fork of the openCFS/cfs and what to get the latest changes from upstream openCFS/CFS repository. For this scenario the master branch from you own fork is used and the upstream master branch. Note that the openCFS master branch is not writable, but usually the one of your own fork.

Verify what URL your local CFS repo's origin refers to (i.e. show details of remotes)

git remote -v

Then add a new remote called openCFS for the upstream CFS repo

git remote add openCFS https://github.com/openCFS/cfs.git # if you only have read permissions to the repo
git remote add openCFS git@gitlab.com:openCFS/cfs.git # if you are a member of the openCFS group

The procedure works equivalently for other repos (Testsuite, ...) - just replace the URLs

Since you now have more than one remotes configured, many git commands need to be more specific, explicitly specifying the remote you want to interact with.

You can now fetch data from the new remote

git fetch openCFS

see if there are difference in the upstream master branch

git log openCFS/master
git diff openCFS/master

or pull data into the local repo's master branch

git checkout master
git pull openCFS

Now your local repo has the new content from the upstream repo. You can push it to your fork

git push # when you are on master
git push origin openCFS/master:master # explicitly push local ref of the upstream master branch (`openCFS/master`) to a branch (`:master`) of the fork remote (`origin`). 

Make sure you have permission to directly push to the master branch in your fork's repo (this might be disabled via the gitlab repository settings)

The syntax is as follows: With git push <repository> <source>:<destination> you push the <source>, usually a local branch or single commit to <destination> in a remote <repository>.

Sometimes you need to be more specific in the destination too, e.g. by giving refs/heads/branch_name instead of just branch_name or refs/tags/my_new_tag. For example, to check out a branch existing on both remotes use

git fetch --all # fetch updates from all remotes
git checkout --track remote_name/branch_name # e.g. origin/myBranch

you should now be on a "new" local branch with the contents existing on the remote.

FAQ

Here are some short answers to a few frequently asked questions. Feel free to extend the list.

My HEAD got detached!

A detached HEAD occurs when you check out a commit that is not the tip of a branch. It is resolved by simply checking out a branch.

When you first clone the project on your local machine you will most likely run into the problem of a detached HEAD regarding the submodule. If this is the case it is most likely that your submodule is not tracking a branch (e.g. master). To fix this you just have to tell the submodule that it should track the master branch via

cd Testsuite
git checkout master # checkout the master branch and set tracking information automatically
git branch -u origin/master master # alternative: explicitly set (upstream) tracking info

For more information see stackoverflow

what does fast-forward merge mean?

Consider the history

A -- B         master
      \
       C -- D  feature

where A and B are contained in the master branch (B is the current HEAD of the master branch), and the feature branch contains commits A-B-C-D (D is the HEAD of the feature branch).

When merging feature into master the merge is fast-foward because the HEAD of the master branch just needs to move (fast-forward) along the common commit history to the HEAD of the feature branch.

what does rebase mean?

It means taking commits and putting them somwhere else in the history ...

Consider the history

A -- B -- E    master
      \
       C -- D  feature

where the master branch consists of 3 commits A-B-E and the feature branch of commits A-B-C-D. When you rebase feature onto master you will get

A -- B -- E           master
           \ 
            C' -- D'  feature

where master stayed unchanged, and the commits in feature are moved. They are now called C' and D' to indicate that the have the same changeset (diff) but different SHA (since their history changed).

You can also combine commits during rebasing.

I did not read the wiki and now I'm stuck on a branch that has severely diverged from the master

If you find yourself in a scenario where you happily developed your stuff but never rebased your work on the current master, you will most likely face the problem that you can't easily rebase your work on the master due to too many merge conflicts. Besides the standard rebase one possibility would be to cherry-pick your changes on a new branch based on the current master. We start by checking the history via

git log

and note the commits we want to cherry-pick. Then we check out the new branch based on the current master

git checkout master
git pull
git checkout -b my_new_branch

Finally, we have to step through our commit history and cherry-pick our commits by

git cherry-pick commit-SHA # cherry-pick a specific commit

or

git cherry-pick SHA1^..SHA2 # cherry pick multiple commits ranging from (and including) SHA1 to SHA2 where SHA1 is older than SHA2

push rejected (non-fast-forward)

The things you want to push would overwrite something on the remote ...

• did you change history (e.g. amend or rebase)?
• are you sure you push to the correct branch?
• maybe someone else was working on the same branch!

Can be resolved by force-push, which will delete/overwrite the remote with your local version. Always try --force-with-lease before --force.

When should I push with force?

Using the force option (-f or --force) for the push command will overwrite the remote with your local version. Always try --force-with-lease before --force.

• required when you have changed the history after you already pushed once
• often needed when you get a merge request in shape (fixing build configs only tested in the pipeline, typos in commit messages, ...)

When you force-push, you might delete other people's work. Therefore,

• check where you force-push, i.e. look at the branch in the web-ui (on gitlab)
• explicitly specify the branch to force push, e.g. git push -f origin +testing (+ means only push this ref), or git push -f origin my-local-branch:remote-merge-request-branch)

Useful commands

This list represents the most useful commands for contributing.

clone

git clone URL # will clone the complete repository

fetch / pull

To get new data from the remote you need to fetch it

git fetch

simply gets new data from the default remote, without changing the state of the working directory.

git pull --rebase # will fetch remote changes and rebase your local changes on top of them

to set the default for pull do

git config [--global] pull.rebase true

branching

Branches are created locally and can be pushed to the remote

git branch # lists branches
git branch <branch-name> # creates branch 
git checkout <branch-name> # switches to branch
git checkout -b <branch-name> # create and checkout branch as a one-liner

staging

Before we can commit something, we need to stage the changes. Here we can stage files individually to separate changes into multiple commits.

git add FILE # stages specific file
git add . # stages all unstaged changes

commit

After staging our changes, we can commit them via

git commit -m 'Commit-message' # one-liner for commiting changes (used for short commit messages < 50 chars)
git commit # opens the editor (useful for longer commit messages)

commit history

If we want to take a look at the commit history, we can use

git log # shows the commit history for the current branch

rebase

While implementing changes the commit history will most likely be cluttered with short commit messages, which need to be collected and documented. With an interactive rebase we can rewrite the history and combine commits and add new commit messages. This is very useful for cleanups before merging. For the rebase we have multiple options. You can specify

• a commit SHA
• how many steps (commmits we want to go back)
• a branch to rebase on (e.g. master)

git rebase -i SHA # interactive rebase starting from the commit with SHA
git rebase -i HEAD~N # interactive rebase for the last N commits
git rebase -i master # rebase our changes on the master branch

reset

If you want to reset your local repository to a specific commit-SHA you can do this via

git reset commit-SHA # specify commit which should be tested

This can be useful if you want to split commits or if you simply want to force-push prior commits to trigger the pipeline for each commit (see CI/CD pipeline).

cherry-pick

Cherry-picking can be used to pick individual commits (or a range of commits) and apply them on a different branch. Hence, the picked commit gets based on top of the chosen branch. For individual commits use

git cherry-pick commit-SHA # cherry-pick a specific commit

and for a range of commits you can use

git cherry-pick SHA1^..SHA2 # cherry pick multiple commits ranging from (and including) SHA1 to SHA2 where SHA1 is older than SHA2

merge

Merge is local in git. Make sure you are on one of the to-be-merged branche and merge the other one now:

git merge <branch-name>

tag

Git tags are simply references to the repository state (SHA1) of any commit. Create and list them with

git tag -a name
git tag -l
git show <tagname>

You can also define aliases to shorten the input (see below).

stash

You can use git stash to temporarlily store all changes and get a clean repo. This is often useful if e.g. pull or checkout would overwrite stuff (and git refuses to do so). With git stash applyyou can apply the stashed changes again.

Tips

Switch on colors

git config --global color.ui auto
git config --global color.diff auto
git config --global color.status auto
git config --global color.branch auto

Set mergetool to vimdiff

git config --global merge.tool vimdiff
git config --global merge.conflictstyle diff3
git config --global mergetool.prompt false

Possible mergetools are meld, opendiff, kdiff3, tkdiff, xxdiff, tortoisemerge, gvimdiff, diffuse, ecmerge, p4merge, araxis, vimdiff, emerge.

Show all branches

git branch -a

List information about repository (e.g. URL)

git remote show origin

Define aliases

git config --global alias.ci 'commit -v -a'
git config --global alias.st 'status'
git config --global alias.co 'checkout'
git config --global alias.info 'remote show origin'

To save password when using https you may modify the remote-url in .git/config to

[remote "http"]
         url = https://username:password@cfs-dev.mdmt.tuwien.ac.at/cfs/CFS.git
         fetch = +refs/heads/*:refs/remotes/http/*

however, this is not recommended -> use SSH.

You can define a shell prompt telling you which branch you are on:

1. download git-prompt.sh and source it in your .bashrc
2. Change your prompt to call __git_ps1, e.g. export PS1='[\u@\h \W$(__git_ps1 " (%s)")]\$ '

gitlab has the option ignore whitespace changes for the diff view. This can be helpful, if someone changed the indentation of a file.
Unfortunately gitlab does not support this for blame. However, you can still checkout the file and perform git blame -w FILE in a terminal.