This repo is now deprecated and will no longer be actively maintained. Why? Because there is finally a community supported edition that is very similar! \o/ You can read the new and improved instructions here: https://github.com/openstack-infra/puppet-openstackci/blob/master/contrib/README.md
If you've used this solution, migrating to the new solution should
be relatively straight-forward. A lot of the 'magic' in the script
has been made manual so that it is more explicit. You will also be
using a better approach to maintaining a masterless puppet environment.
In general, many of the custom
values from the
vars.sh are now stored in a hiera yaml file.
In addition, all the configuration files stored in os-ext-testing-data
repo are now available in
Future bug fixes, pull requests, etc. should be done on the above mentioned repositories for the benefit of the larger Openstack community and even some outside of OpenStack.
OpenStack External Test Platform
!! THIS REPOSITORY IS VERY MUCH A WORK IN PROGRESS !!
PLEASE USE AT YOUR OWN RISK AND PROVIDE FEEDBACK IF YOU CAN!
This repository contains documentation and modules in a variety of configuration management systems that demonstrates setting up a real-world external testing platform that links with the upstream OpenStack CI platform.
It installs Jenkins, Jenkins Job Builder (JJB), the Gerrit Jenkins plugin, Nodepool, and a set of scripts that make running a variety of OpenStack integration tests easy.
Currently only Puppet modules are complete and tested.
Background reading: third_party
The links below contain some out of date information:
NEW 7/1/2015: This repo is being migrated to use project-config and puppet-openstackci
This 3rd party ci repo is in the process of being migrated to use the [common-ci approach] (http://specs.openstack.org/openstack-infra/infra-specs/specs/openstackci.html)
As part of that, there will be a migration from the os-ext-testing-data config repo to use a repo following the structure of [project-config] (https://github.com/openstack-infra/project-config/).
Fortunately, that task is simple:
Create a new repo called e.g. project-config-ci-name
cp ~/os-ext-testing-data/etc/zuul/layout.yaml ~/project-config-ci-name/zuul/
cp ~/os-ext-testing/puppet/modules/os_ext_testing/files/zuul/openstack_functions.py ~/project-config-ci-name/zuul/
update os-ext-testing-data/vars.sh to include export PROJECT_CONFIG=http://your_git_url/project-config-ci-name.git
Push the changes. They'll be checked out in /etc/project-config
NEW 7/17/2015 - Now using common-Jenkins Job Builder
cp -R ~/os-ext-testing-data/etc/jenkins_jobs/config/* ~/project-config-ci-name/jenkins/jobs
Push the changes. They'll be checked out in /etc/project-config
NEW 8/24/2015 - Migrate nodepool configuration files to project-config
There are a few big changes here. First, previously, the nodepool.yaml file was a template where some portions where populated by puppet. After migrating, the nodepool.yaml will be a static file containing all usernames and credentials.
Second, previously the nodepool elements and scripts used to build nodepool images were copied from http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool. The puppet scripts allowed you to override and add additional scripts/elements in your os-ext-testing-data repository. This is no longer supported. Instead, you manually fork the scripts and elements and maintain them separately.
If you already have a nodepool.yaml file create previously, copy it from /etc/nodepool to ~/project-config-ci-name/nodepool otherwise, create a new one taking care to ensure all values are fully resolved. [Nodepool Configuration Manual] (http://docs.openstack.org/infra/nodepool/configuration.html)
If you already have scripts/elemements in /etc/nodepool, copy them over to ~/project-config-ci-name/nodepool/elements and ~/project-config-ci-name/nodepool/scripts. Otherwise, start with the scripts/elements provided [by upstream's project config] (http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool ) and adjust to make them work in your environment.
Remove any remaining values in your previous os-ext-testing-data/vars.sh such as PROVIDER_.*
If you need help, you can:
Submit a question/issue via github
Ask in the third party ci meetings
Ask in the mailing list. Use [third-party] tag in the subject.
Ask on IRC freenode in channel #openstack-infra
The following are pre-requisite steps before you install anything:
Read the official documentation: http://ci.openstack.org/third_party.html
Get a Gerrit account for your testing system registered
Ensure base packages installed on your target hosts/VMs
Set up your data repository
Below are detailed instructions for each step.
Registering an Upstream Gerrit Account
You will need to register a Gerrit account with the upstream OpenStack CI platform. You can read the instructions for doing that
Ensure Basic Packages on Hosts/VMs
We will be installing a Jenkins master server and infrastructure on one host or virtual machine and one or more Jenkins slave servers on hosts or VMs.
On each of these target nodes, you will want the base image to have the
ca-certificates packages installed before
running anything in this repository.
Set Up Your Data Repository
NOTE: This section is a out-dated because of the migration towards the common-ci solution & project-config. See those details at the top of this README.
You will want to create a Git repository containing configuration data files -- such as the Gerrit username and private SSH key file for your testing account -- that are used in setting up the test platform.
The easiest way to get your data repository set up is to make a copy of the example repository I set up here:
and put it somewhere private. There are a few things you will need to do in this data repository:
Copy the private SSH key that you submitted when you registered with the upstream OpenStack Infrastructure team into somewhere in this repo.
If you do not want to use the SSH key pair in the
os-ext-testing-dataexample data repository and want to create your own SSH key pair, do this step.
Create an SSH key pair that you will use for Jenkins. This SSH key pair will live in the
/var/lib/jenkins/.ssh/directory on the master Jenkins host, and it will be added to the
/home/jenkins/.ssh/authorized_keysfile of all slave hosts::
ssh-keygen -t rsa -b 1024 -N '' -f jenkins_key
Once you do the above, copy the
jenkins_key.pubfiles into your data repository.
Copy the vars.sh.sample to vars.sh and open up
vars.shin an editor.
Change the value of the
$UPSTREAM_GERRIT_USERshell variable to the Gerrit username you registered with the upstream OpenStack Infrastructure team as detailed in these instructions
Change the value of the
$UPSTREAM_GERRIT_SSH_KEY_PATHshell variable to the relative path of the private SSH key file you copied into the repository in step #2.
For example, let's say you put your private SSH key file named
mygerritkeyinto a directory called
sshwithin the repository, you would set the
If for some reason, in step #2 above, you either used a different output filename than
jenkins_keyor put the key pair into some subdirectory of your data repository, then change the value of the
vars.shto an appropriate value.
Copy etc/nodepool/nodepool.yaml.erb.sample to etc/nodepool/nodepool.yaml.erb. Adjust as needed according to docs: http://ci.openstack.org/nodepool/configuration.html.
Update etc/zuul/layout.yaml according to docs: http://ci.openstack.org/zuul/zuul.html#layout-yaml
Setting up the Jenkins Master
On the machine you will use as your Jenkins master, run the following:
wget https://raw.github.com/rasselin/os-ext-testing/master/puppet/install_master.sh bash install_master.sh
The script will install Puppet, create an SSH key for the Jenkins master, create self-signed certificates for Apache, and then will ask you for the URL of the Git repository you are using as your data repository (see Prerequisites #3 above). Enter the URL of your data repository and hit Enter.
Puppet will proceed to set up the Jenkins master.
Manual setup of Jenkins scp 1.9 plugin
Version 1.8 is publicly available, but does not have all features (e.g. copy console log file, copy files after failure, etc.). Follow these steps to manually build and install the scp 1.9 plugin:
- Download http://tarballs.openstack.org/ci/scp.jpi
- Jenkins Manage Plugins; Advanced; Upload Plugin (scp.jpi)
Restart Jenkins to get the plugins fully installed
sudo service jenkins restart
Load Jenkins Up with Your Jobs
Run the following at the command line:
sudo jenkins-jobs --flush-cache update /etc/jenkins_jobs/config
sudo service zuul start sudo service zuul-merger start
After Puppet installs Jenkins and Zuul and Nodepool, you will need to do a couple manual configuration steps in the Jenkins UI.
Go to the Jenkins web UI. By default, this will be
Manage Jenkinslink on the left
Scroll down until you see "Gearman Plugin Config". Check the "Enable Gearman" checkbox.
Click the "Test Connection" button and verify Jenkins connects to Gearman.
Scroll to "ZMQ Event Publisher" and select "Enable on all Jobs". Double-check the port matches the URL configured for "zmq-publishers" in
Scroll down to the bottom of the page and click
At the command line, do this::
sudo service zuul restart
Running jobs on Jenkins Master
Currently it seems that running jobs on Jenkins Master directly no longer works. It seems to be a regression with newer versions of Jenkins. So skip that and go straight to:
Setting up Nodepool Jenkins Slaves
Re-run the install_master.sh script for your changes to take effect.
Make sure the jenkins key is setup in the 'cloud' provider with name "jenkins". TODO: make it configurable.
Manually create your first image. This is helpful to debug errors. On subsequent debug runs, consider enabling DIB_OFFLINE=true mode to save time. Remember to unset DIB_OFFLINE when creating the real image.
See here for more information. [project-config DIB tips] (https://github.com/openstack-infra/project-config/tree/master/nodepool/elements)
sudo su - nodepool #optional export DIB_OFFLINE=true nodepool image-build <image-name>
sudo service nodepool start # Or manually (in a screen session): sudo su - nodepool source /etc/default/nodepool nodepoold -d $DAEMON_ARGS
Setting up Log Server
The Log server is a simple VM with an Apache web server installed that provides http access to all the log files uploaded by the jenkins jobs. It is a separate script because the jenkins-zuul-nodepool 'master' server may/can not be publicly accessible for security reasons. In addition, separating out the log server as its own server relaxes the disk space requirements needed by the jenkins master.
Installing the Log Server on the same VM as Jenkins/Nodepool/Zuul is not supported.
It's configuration uses the puppet-openstackci scripts, which provide the friendly log filtering features, hightlighting, the line references, etc.
For simplicity, it is recommended to use the same jenkins key for authentication.
wget https://raw.githubusercontent.com/rasselin/os-ext-testing/master/puppet/install_log_server.sh export DOMAIN=your.domain.com export JENKINS_SSH_PUBLIC_KEY=/full/path/to/public/key #MANUALLY Update the LOG_SERVER_DOMAIN & JENKINS_SSH_PUBLIC_KEY_CONTENTS variables bash install_log_server.sh
When completed, the jenkins user will be able to upload files to /srv/static/logs, which Apache will serve via http. This is accomplished by adding publishers to your jenkins job.