Fetching contributors…
Cannot retrieve contributors at this time
428 lines (340 sloc) 19.1 KB
This file documents dev, the workflow helper tool for Crowbar
dev is designed to help automate the workflow and infrastructure
related tasks associated with development in general with the Crowbar
git tree layout. Dev is designed around the following assumptions:
* Everyone has their own forks on Github of the Crowbar repository
and all of the barclamps.
* You do not have to manually keep submodule references up to date.
* Your development workflow will involve regular synchronization
against the dellcloudedge repositories.
* The only path for getting code into the dellcloudedge repositories
is via pull request.
* A checkout of Crowbar.
* Bash 4, ruby, rubygems, and the json gem.
If running under Windows, make sure that all these operate in your
cygwin or msysgw enviromnent.
* A github username and password.
Day to Day Workflows:
Initial Setup:
1: Clone the Crowbar repository from
2: Run crowbar/dev setup.
a: Provide your github login ID and password. Dev will need it to
to handle takling to Github.
b: dev will create any missing forks of the dellcloudedge Crowbar
c: dev will add a remote named personal to each forked repository.
d: dev will add git config entries describing how local changes
will be backed up to each repository.
3: Create your local build cache. See for more
4: Re-run setup any time you need to pull in a new barclamp, or if
you need to use non-Github remotes for additional functionality.
Regular Development:
1: Run dev fetch followed by dev sync to fetch and merge changes from
the dellcloudedge repositories.
a: Dev will fetch all changes from all upstream remotes for all
b: Dev will attempt to synchronize all the local branches with
their corresponding remote branches. If there are merge
conflicts, the sync process will stop at the branch that was
having problems. From there, you can manually fix the
conflicts and rerun dev sync.
2: Run dev new-feature <featurename> to start working on a new
a: This command shold be run from the development release.
b: Dev will create branches in the feature/<featurename> namespace
in the main Crowbar repository and in all the barclamps. The
initial starting point for these branches will be the tips of
the corresponding branches in the development namespace.
3: Hack/build/test/commit.
4: Run dev backup to back up your changes. This force-pushes your
changes to your personal forks of the crowbar repositories on
Github, or creates new personal branches for non-Github upstreams.
5: If you are not ready to create a pull request for your changes, go
to 1.
Ready for pull request:
1: Run dev pull-requests-prep
a: dev will verify that all the local Crowbar repositories are
b: dev will perfom a fetch and a sync, and abort if there were any
merge conflicts.
c: dev will figure out what barclamps and what main Crowbar
branches are candidates for a pull request in the current release.
d: dev will print a command line wil all the branches and
barclamps that are candidates for pull requests in the current release.
2: Run dev pull-requests-gen
a: dev will ask for a title to be used for all the pull requests.
b: dev will open an editor for you to type in a body that will be
used for all the pull requests that will be generated.
c: dev will work out the proper order that a reviewer will need to
merge the pull requests in based on barclamp submodule and
branch dependencies.
d: If submodules are being updated, dev will create the commit
records needed to point the main Crowbar branches at the proper
submodule commits.
e: dev will issue pull requests in the order worked out
earlier. Each individual pull request will have a sequence
number added to the title indicating the proper merge order.
f: Github will email the review team at Dell with each pull
Review pull request:
1: Dell reviewer is notified of the pull request by Github.
2: Reviewer reviews and tests pull requests as a group.
3: If the changes are OK to be pulled, reviewer merges them in order.
Release Branching Structure:
The dev tool has the notion of a release, which it manages in terms of
related branches. Currently implemented release types are:
* Development which consists of the top-level branches tracked
branches in the Crowbar repository (master and its descendents),
and the master branch in each of the barclamps.
Development is where the day-to-day development on new features and
bugfixes happens.
* Releases, which consist of branches prefixed with release/<release
name>/ in the main Crowbar repository and in the
barclamps. Releases are named with a unique name (similar to Ubuntu
or Debian release names), and only get stabilization and bugfix updates.
* Feature bundles, which consist of branches prefixed with feature/<feature
name>/ in the main Crowbar repository and in the barclamps.
Features are local to your private forks and personal branches, and
get merged the main dellcloudedge repositories via pull requests to
the appropriate development branch in the appropriate dellcloudedge
repository. For now, feature branches should always be based on and
merged back into the development release.
* Stable, which consists of branches prefixed with stable/ in the
main Crowbar repository and in the barclamps. The details of how
the stable release will work is TBD.
Release Workflows:
Getting a list of known releases:
1: Run dev releases
Getting the release you are currently on:
1: Run dev release
Switching to a different release:
1: Run dev switch <release name>
a: dev will verify that the crowbar repositories are "clean", and
it will refuse to do anything if they are not.
b: dev will checkout the appropriate master branch for the release
in all the barclamps.
c: If there is a branch that matches the one you are on in the
main Crowbar repository, dev will check that branch out,
otherwise it will check out the master branch in that release.
Cutting a new release:
1: Ensure that all the crowbar repositories are in the exact state
you want the new release to start out in.
2: Run dev cut_release <new release name>.
a: dev will verify that the release name is not already in use in
the current repository.
b: dev will create the master branch for the new release in each
of the barclamps.
c: dev will create a branch structure for the new release based
on the branch structure of the current release.
Managing non-dellcloudedge repositories:
The dev script has the capability of integrating local or private
forks of Crowbar into its workflow. To do this, you will need to
add the following pieces of information to your $HOME/.build-crowbar.conf:
* Add any local branches to the DEV_BRANCHES hash. This hash defines
the parent -> child relationships between branches:
Add one entry for each additional branch you want to track. If a
branch has itself for a parent it will be considered to be the root
of a new branch hierarchy.
* Add an entry in DEV_REMOTE_SOURCES for each additional remote
you want the dev script to consider to be an upstream. This hash
has the following structure:
The dev script expects to find a fork of the main Crowbar
repository there, along with repositories for any extra barclamps
you are pulling in as submodules.
* Add an entry in DEV_REMOTE_BRANCHES to describe what additional
branches each remote should be considered the "canonical" upstream
of. Each entry should have the following structure:
DEV_REMOTE_SOURCES["remote"]="branch another_branch"
* Append your new remotes to the DEV_REMOTES array. This array
defines the order in which origins are consulted when performing
fetch and backup operations
Once you have added all the extra data you need, rerun dev setup and
the script will fetch everything needed to satisfy the dependencies
that the new settings imply.
For example, the internal Dell branches of Crowbar need the following
additional config settings in $HOME/.build-crowbar.conf:
DEV_REMOTE_BRANCHES["dell"]="openstack-build hadoop-build"
dev currently assumes that origin in the main Crowbar barclamp will
always point at the dellcloudedge repositories, that the remote named
personal will always point at your github forks, and that it can
control the personal/<your_github_id>/ branch namespace upstream
remotes for any non-github repositories.
If your working circumstances involve not always having access to all
the remotes you have configured, you can set the DEV_AVAILABLE_REMOTES
environment variable to a list of remotes that are available. dev
will only try to operate on remotes that it currently thinks are
reachable, but things will probably not work too well if origin gets
Dev commandline options:
setup: Perform initial setup for the Crowbar repositories
dev setup prepares the repositories to be managed by the dev
tool. It handles caching your github credentials, ensuring that
all of the barclamps get cloned and configured properly for fetch,
sync, and backup operations. It can be re-run nondestructivly if
new barclamps are added or your local development environment
needs to change.
is_clean: Tests to see if the crowbar repo trees are clean.
Without any options, if any of the trees have uncommitted
modifications to files under revision control, or there are any
files present that are not under revision control that are not
also ignored and that are not parts of a barclamp, this command
will fail. This command ignores submodule reference mismatches by
With the --barclamps option, dev is_clean will not ignore
submodule reference mismatches.
With the --path option, dev is_clean will just check the locations
you pass on the commandline for cleanliness.
fetch: Fetch updates to Crowbar and the barclamps from upstream.
This command fetches updates for the main Crowbar repository and
all the barclamps from all of the upstream repositories listed in
the DEV_REMOTES array. It does not make any modifications to the
working trees.
sync: Sychronize local branches with their upstreams and local edits.
This command assumes that remote changes have already been fetched
into your local repositories using dev fetch. Assuming that has
been done, dev sync will perform the following operations:
1: For each barclamp that is considered to be part of the current
release, sync all the changes from the origin remote with the
corresponding local branches for that barclamp. If any merge
conflicts are reported, dev will remember them for later.
2: For each local branch in the Crowbar repository, sync that
branch with the proper upstream branch. If any merge conflicts
are detected, dev will exit with an error message.
3: Ripple changes out from parent to child branches for this
dev sync may use either a rebase or a merge to synchronize a local
branch with its corresponding upstream. This decision is made on
a per-branch basis, and the algorithm that it uses is:
1: Create a temporary branch that holds the state of the current
branch we are looking at.
2: Merge the upstream branch into the branchwe are looking at. If
that fails, we had a merge conflict which the user will have to
resolve, and we abort.
3: Switch over to the temporary branch, and rebase it onto its
upstream branch. If the rebase fails for any reason, abort it can
decide to use the merged branch.
4: Compare the working trees for the merged branch and the rebased
branch. If they are identical, use the rebased branch. If they
are not, use the merged branch.
backup: Backs up local commits to Github forks or personal branches.
Use this command whenever you want to make sure your local commits
are backed up to locations outside your local repositories. By
default, any branches in the main Crowbar repository or any
branches in barclamps that originally came from the dellcloudedge
repositories will be pushed to your forks of those repositories, and any
branches in the main Crowbar repository or any branches in
barclamps that came from somewhere else will be pushed to branches
in the personal/github_login/ namespace in the proper upstream
repostitories for that branch or barclamp.
push: Unconditionally push a branch to the personal remote.
Use this command in the main Crowbar repository whenever you want
to unconditionally push it to your fork on Github. Any arguments
are interpreted as branches to push, and if there are no arguments
the current branch is pushed.
pull-requests-prep: Prepare the local repository to issue a pull request.
This command will perform a fetch, a sync, and a backup
operation. If all those succeed, the command will then figure out
what all barclamps and Crowbar branches in the current release
have local commits, and it will print out a command line that can
be used to have dev actually generate the pull requests.
If the Crowbar repositories are in a feature release,
pull-requests-prep will figure out what is needed to merge the
feature branches into the development release.
pull-requests-gen: Generate pull requests.
This command creates a series of pull requests based on the
command line arguments passed. It will automatically include
updated submodule references for the barclamps if needed, and if
just barclamps are passed it will also figure out what branches in
the current release need updates and generate requests for them as
If the Crowbar repositories are in a feature release, the pull
requests will be generated against the corresponding branches in
the development release. This enables feature releases and pull
requests to form the basis of gated-trunk style development.
release: Shows the current release of Crowbar based on git branches.
releases: Show all the releases that dev knows how to work on.
switch: Switch from one release to another.
This command checks out the appropriate release branch in all the
barclamps for the release you are switching to, and then checks out
the branch that most closely corresponds to your current branch in
the release you are switching to. If you do not pass a release, it
will check out everytthin for the current release instead.
build: Build a release of Crowbar.
Despite the name, build mainly concerns itself with setting the Crowbar
repositories up before invoking to perform the actual
build. The build command takes the following parameters:
--os = operating system to stage Crowbar on to.
--release = Release of Crowbar to build.
--branch = Branch within a release of Crowbar to build
--exact = If present, dev will run git submodule update after processing
the --release and --branch options, and the --no-cache-update
and --no-metadata-update parameters will be passed to If not present, the submodules will be
checked out to the appropriate master branch for the release.
Any other parameters will be passed unchanged to
See for more details.
If your build cache is managed by Git, and it has the same branching
structure as the Crowbar releases, dev will switch to the appropriate
branch in the build cache before invoking This makes
it easier to have seperate releases with known working sets of packages.
cut_release: Cuts a new release based on the current state of Crowbar.
Do not use this command unless you really know what you are doing.
new-feature: Creates feature branches for a new feature.
Features are the primary method of working on new
features/bugfixes/quasi-independent changes in
Crowbar. They allow for gated-trunk style development in Crowbar
by treating the development release as the trunk, and the feature
releases as things that can be merged into the trunk via
pull-requests on Github.
Internally, features are implemented as a release type,
so any dev command that works on releases will also work on
erase-feature: Erases feature branches created with new-feature.
This command is intended to be run once a feature had been merged
into the main development release -- once a feature has been
merged, there is generally no point in keeping the branches around
to clutter up your local git branches.
erase-feature will check to see of the master branch of the
feature branches has been merged with the master of the
development release, and it will exit with an error if there are
changes in the feature that have not been merged.
branches: Show the branches that are part of the current release
find-parent: Find the best logical parent of a feature bundle.
This command tries to find the release that has the smallest
number of commits that would have to be examined in order to merge
the feature into it.
reset-release: Reset a release or feature bundle.
This command allows you to reset a release or feature bundle to
the exact state it was in at either the last backup or from the
upstream repositories. It takes the following options:
--release = The release or feature you want to reset.
--target = The state you want the release to be reset to.
This can be either upstream (which will reset the branches to
their corresponding remotes), or backup (which will reset the
branches to their last backup).
--skip-master - If present, this will cause reset-release to leave
the master branch in the release alone. It is implicitly set if
you are resetting the current release.
--skip-barclamps - If present, this will cause reset-release to
leave the barclams alone.
If run without any options, it will all
the branches for the current release to the state they were in as
of the last backup, except for the master branch of the current
release in the crowbar repository, which will be left alone.
scrub-merged-pulls: Scrub tracking branches for pull requests
Normally, you should not have to run this command, because it is
implicitly run after every fetch. However, if you have pull
requests that were not merged in for whatever reason, your
personal remote will slowly accumulate these branches. To clear
them all out, you can add the --all parameter, which will
unconditionally delete all the pull request tracking branches.