Upgrade testing of OpenStack projects built on top of DevStack.
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
data Generate plugin list for registry document Mar 1, 2018
doc/source Generate plugin list for registry document Mar 1, 2018
inc Leverage systemd for service running check Aug 30, 2017
playbooks/legacy Add grenade-py3 job Nov 14, 2018
projects Merge "Create config file before database migrate" Dec 18, 2018
tools Generate plugin list for registry document Mar 1, 2018
.gitignore Generate plugin list for registry document Mar 1, 2018
.gitreview add .gitreview to the project Mar 26, 2013
.zuul.yaml Add grenade-py3 job Nov 14, 2018
HACKING.rst Use git.openstack.org everywhere Dec 30, 2015
LICENSE Initial repo setup Aug 3, 2012
PLUGINS.rst Document when pre-release upgrade scripts are run Sep 6, 2018
README.rst Document when pre-release upgrade scripts are run Sep 6, 2018
TODO.rst Remove exercises and run tempest smoke tests instead Feb 4, 2014
cache_git.sh Remove ironic code from tree Dec 11, 2015
clean.sh Only call save_mysql_dbs if using mysql Sep 18, 2018
devstack.local.conf.base Update devstack keys from config files Jan 26, 2017
devstack.local.conf.target Ensure we use uwsgi for everything Sep 5, 2017
devstack.localrc.base Update devstack keys from config files Jan 26, 2017
devstack.localrc.example Updating devstack.localrc.example for accuracy Jan 25, 2017
devstack.localrc.target Update devstack keys from config files Jan 26, 2017
functions Only call save_mysql_dbs if using mysql Sep 18, 2018
grenade.sh Merge "Remove confusing mention of WSGI_MODE" Aug 30, 2018
grenaderc Merge "Initialize ERROR_ON_CLONE flag in grenaderc" Apr 7, 2017
prep-target Add WSGI_MODE setting for target Apr 17, 2017
run_resource.sh add convenience tools Apr 24, 2015
save-state Fix broken logic for detecting if executable exists Nov 12, 2016
setup-grenade Use git.openstack.org everywhere Dec 30, 2015
setup.cfg Add docs building support to grenade Aug 28, 2015
setup.py Add docs building support to grenade Aug 28, 2015
test-requirements.txt Add a venv tox job Nov 29, 2017
tox.ini Generate plugin list for registry document Mar 1, 2018
upgrade-tempest Fix grenade runs which don't use tempest Dec 22, 2016

README.rst

Grenade

Grenade is an OpenStack test harness to exercise the upgrade process between releases. It uses DevStack to perform an initial OpenStack install and as a reference for the final configuration. Currently Grenade can upgrade Keystone, Glance, Nova, Neutron, Cinder, Swift, and Ceilometer in their default DevStack configurations.

Goals

Grenade has the following goals:

  • Block unintentional project changes that would break the Theory of Upgrade. Most Grenade fails that people hit are of this nature.
  • Ensure that upgrading a cloud doesn't do something dumb like delete and recreate all your servers/volumes/networks.
  • Be able to grow to support additional upgrade scenarios (like sideways migrations from one configuration to another equivalent configuration)

Theory of Upgrade

Grenade works under the following theory of upgrade.

  • New code should work with old configs

    The upgrade process should not require a config change to run a new release. All config behavior is supposed to be deprecated over a release cycle, so that upon release new code works with the last releases configs. Those configs may create deprecation warnings which need to be addressed before the next release, but they should still work and largely have the same behavior.

  • New code should need nothing more than 'db migrations'

    Clearly the release of new code may include new database models. Standard upgrade procedure is to turn off all services that touch the database, run the db migration script, and start with new code.

  • Resources created by services before upgrade, should still be there after the system is upgraded

    When upgrading Nova you expect all your VMs to still function during the entire upgrade (whether or not Nova services are up). Taking down the control plane should not take down your VMs.

  • Any other required changes on upgrade are an exception and must be called out in the release notes.

    Grenade supports per release specific upgrade scripts (from-juno, from-kilo). These are designed to support upgrades where additional manual steps are needed for a specific upgrade (i.e. from juno to kilo). These should be used sparingly.

    The Grenade core team requires the following before landing these kinds of changes:

    • The Release Notes for the release where this will be required clearly specify these manual upgrade steps.
    • The PTL for the project in question has signed off on this change.

    Note

    While we expect the various deployment projects within the OpenStack ecosystem, for example TripleO, Kolla, etc, to read the release notes of each project, it is good practice to communicate any exceptional upgrade changes made to Grenade to those teams directly or at least via the openstack-dev mailing list.

Status

Grenade is now running on every patch for projects that support upgrade. Gating Grenade configurations exist for the following in OpenStack's CI system.

  • A cloud with nova-network upgraded between releases
  • A cloud with neutron upgraded between releases
  • A cloud with nova-network that upgrades all services except nova-compute, thus testing RPC backwards compatibility for rolling upgrades.

Basic Flow

The grenade.sh script attempts to be reasonably readable, so it's worth looking there to see what's really going on. This is the super high level version of what that does.

  • get 2 devstacks (base & target)
  • install base devstack
  • perform some sanity checking (currently tempest smoke) to ensure this is right
  • allow projects to create resources that should survive upgrade - see projects/*/resources.sh
  • shut down all services
  • verify resources are still working during shutdown
  • upgrade and restart all services
  • verify resources are still working after upgrade
  • perform some sanity checking (currently tempest smoke) to ensure everything seems good.

Terminology

Grenade has two DevStack installs present and distinguished between then as 'base' and 'target'.

  • Base: The initial install that will be upgraded.
  • Target: The reference install of target OpenStack (maybe just DevStack)

Directory Structure

Grenade creates a set of directories for both the base and target OpenStack installation sources and DevStack:

$STACK_ROOT
 |- logs                # Grenade logs
 |- save                # Grenade state logs
 |- <base>
 |   |- data            # base data
 |   |- logs            # base DevStack logs
 |   |- devstack
 |   |- images          # cache of downloaded images
 |   |- cinder
 |   |- ...
 |   |- swift
 |- <target>
 |   |- data            # target data
 |   |- logs            # target DevStack logs
 |   |- devstack
 |   |- cinder
 |   |- ...
 |   |- swift

Dependencies

This is a non-exhaustive list of dependencies:

  • git
  • tox

Install Grenade

Get Grenade from GitHub in the usual way:

git clone git://git.openstack.org/openstack-dev/grenade

Optional: running grenade against a remote target

There is an optional setup-grenade script that is useful if you are running Grenade against a remote VM from a local laptop.

Grenade knows how to install the current master branch using the included setup-grenade script. The arguments are the hostname of the target system that will run the upgrade testing and the user for the target system:

./setup-grenade [testbox [testuser]]

If you are running Grenade on the same maching you cloned to, you do not need to do this.

Configuration

The Grenade repo and branch used can be changed by adding something like this to localrc:

GRENADE_REPO=git@github.com:dtroyer/grenade.git
GRENADE_BRANCH=dt-test

If you need to configure your local devstacks for your specific environment you can do that by creating devstack.localrc. This will get appended to the stub devstack configs for BASE and TARGET.

For instance, specifying interfaces for Nova is a common use of devstack.localrc:

FLAT_INTERFACE=eth1
VLAN_INTERFACE=eth1

Run the Upgrade Testing

./grenade.sh

Read grenade.sh for more details of the steps that happen from here.