Continuous integration tool for OME projects
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
ansible Pull from Galaxy Aug 3, 2018
docker Bump docker slave to 0.4.0 Mar 7, 2018
git
home Merge pull request #102 from sbesson/mail Aug 7, 2018
jenkins Update plugins Jul 27, 2018
nginx
server Bump to devslave 0.4.0 Mar 3, 2018
slave Bump to devslave 0.4.0 Mar 3, 2018
web Bump to devslave 0.4.0 Mar 3, 2018
.env Also allow to enforce the SSL/TCP ports of the OMERO server container Jun 14, 2018
.gitignore Ignore vendor subdirectory Nov 1, 2017
.travis.yml
CHANGELOG.md Move out update section Oct 21, 2017
README.md Replace BIOFORMATS-maven by BIOFORMATS-build Apr 9, 2018
Troubleshooting.md Add entry for merge command configuration Nov 17, 2017
backup.sh Initial research Apr 30, 2015
bak Add helper scripts Apr 30, 2015
docker-compose.yml
ex
ip builds: being to split jobs Jun 23, 2015
rename.py Cleanup environment variables Mar 1, 2018
requirements.txt
runtest.sh Remove check and fix typo Dec 16, 2017
sslcert update ssl cert script Aug 2, 2016

README.md

Getting started

Devspace is a Continuous Integration tool managed by Jenkins CI providing an automation framework that runs repeated jobs. The default deployment initializes a Jenkins CI master with a predefined set of jobs.

Running and maintaining Devspace requires brief understanding of:

Running and maintaining Devspace in OpenStack requires, in addition, brief understanding of:

Running Devspace requires access to SSH and Git configuration files used for fetching and pushing the Git repositories.

Devspace code depends on the following repositories:

and for OpenStack (optional)

Installation

You can either deploy manually a devspace on a Docker host or you can use the Ansible playbooks to deploy a devspace on OpenStack.

Deploy on a Docker host

The following instructions explain how to deploy a devspace on a Docker host.

  • Log into the Docker host using ssh

  • Install the prerequisites Docker engine and Docker compose either globally or in a virtual environment:

    $ pip install docker-compose
    
  • Create a directory /data/username and change ownership:

    $ sudo mkdir /data/username
    $ sudo chown username /data/username
    
  • Clone the devspace Git repository:

    $ git clone https://github.com/openmicroscopy/devspace.git
    $ cd devspace
    
  • Generated self-signed SSL certificates for the Jenkins and NGINX containers:

    $ ./sslcert jenkins/sslcert HOST_IP
    $ ./sslcert nginx/sslcert HOST_IP
    

    alternatively put your own certificate .crt and .key in the above locations.

  • Copy the SSH and Git configuration files used for fetching and pushing the Git repositories under slave/.ssh and slave/.gitconfig. This is usually your own SSH and Git configuration files

  • Run rename.py to match your topic name. Specify the Git user corresponding to the confguration files used above. If you do not yet have topic branches available on origin, use "develop" or one of the main branches:

    $ ./rename.py MYTOPIC --user git_user
    
  • This will also replace the USER_ID of the various Dockerfile with the ID of the user who will run the devspace, assumed to be: id -u, i.e. the current user.

  • Set environment variables in .env, especially:

    JENKINS_USERNAME=devspace
    JENKINS_PASSWORD=<password>
    
  • Optionally, commit all the deployment changes above on the local clone of the devspace repository.

Start and configure:

  • Start devspace using docker-compose:

    $ docker-compose up -d
    

    By default, this will use the name of the directory as the project name. In the case of a shared Docker host, it is possible to override the project name using

    $ docker-compose up -p my_project -d
    
  • Retrieve the dynamic port of the Jenkins NGINX container. You can access the Jenkins UI from https://HOST_IP:PORT after accepting the self-signed certificate:

    $ docker-compose -p my_project port nginxjenkins 443
    
  • [Optional] Turn on Basic HTTP authentication for Jenkins

    sudo htpasswd -c jenkins/conf.d/passwdfile nginx 
    

    and update jenkins/conf.d/jenkins.conf:

    auth_basic "Restricted";
    auth_basic_user_file /etc/nginx/conf.d/passwdfile;
    

Deploy on OpenStack

The following instructions explain how to deploy a devspace on OpenStack. First, you will need to have an account on OME OpenStack.

The SSH and Git configuration files are used for fetching from and pushing to the Git repositories. They will be copied to the devspace.

The OpenStack and SSH and Git following steps are only need to be done the first time you generate instances.

OpenStack configuration

  • Log into OpenStack
  • Register a key:
    • Go to the Access & Security tab
    • Click on Key Pairs and then on Import Key Pair
    • Copy the content of the public key you use to access our resources e.g. id_rsa.pub
    • The name you used is referred below as your_openstack_key
  • Download your configuration:
    • Go to the Access & Security tab
    • Then API Access
    • Click on Download OpenStack RC File v2.0
    • The file will be named by default omedev-openrc.sh. It will be used to set environment variables needed to connect to OpenStack via the command line.

SSH and Git configuration files

In order to be able to push result of the build job to your GitHub account, you will need a SSH key without passphrase. The key must be named id_gh_rsa. The key and the configuration files will be copied to the devspace.

  • Create a directory devspace_config where you wish and a directory devspace_config/.ssh

  • Generate a SSH key without passphrase in devspace_config/.ssh directory:

      $ ssh-keygen -t rsa -b 4096 -C "your_email_address" -f path/to/devspace_config/.ssh/id_gh_rsa -q -P ""
    
  • Upload the corresponding public key i.e. id_gh_rsa.pub to your GitHub account

  • Create a file devspace_config/.ssh/config and add the following:

      Host github.com
          User git
          IdentityFile ~/.ssh/id_gh_rsa
    
  • Create a devspace_config/.gitconfig file

  • Generate a GitHub token and add it to devspace_config/.gitconfig. Minimally the file should contain:

      [github]
              token = your_token
              user = your_github_username
      [user]
              email = your_email_address
              name = your_real_name
    
  • The devspace_config directory should look like:

/path/to/devspace_config/
   .gitconfig
   .ssh/
        config
        id_gh_rsa
        id_gh_rsa.pub

Create and provision the devspace

  • Clone the devspace Git repository:

      $ git clone https://github.com/openmicroscopy/devspace.git
    
  • Create a virtual environment and from the devspace directory, install shade to access OpenStack via the command line and Ansible:

      $ virtualenv ~/dev
      $ . ~/dev/bin/activate
      $ cd devspace
      (dev) $ pip install -r requirements.txt
    
  • Source the OpenStack configuration file to set the environments variables allowing connection to OpenStack via the command line, adjust to your local configuration:

      (dev) $ . path/to/omedev-openrc.sh
      Enter your password
    

The following commands need to be executed from the ansible subdirectory.

  • Install the various ansible roles from the Galaxy website:

      (dev) $ cd ansible
      (dev) $ ansible-galaxy install -r requirements.yml
    

To "upgrade" roles, you may want to specify --force when installing the roles.

  • Create an instance on OpenStack using the playbook create-devspace.yml. It is recommended to prefix the name of the devspace by your name or your initals:

      (dev) $ ansible-playbook create-devspace.yml -e vm_name=your_name-devspace-name -e vm_key_name=your_openstack_key
    

By default the size of the volume is 50GiB, if you required a larger size, it can be set by passing for example -e vm_size=100. The Floating IP of the generated instance is referred as devspace_openstack_ip below.

  • To provision the devpace, use the playbook provision-devspace.yml. Before running the playbook you will minimally need to edit the value of the parameter:
    • configuration_dir_path: set it to path/to/devspace_config

See ansible-role-devspace for a full list of supported parameters. Provision the devspace by running:

    (dev) $ ansible-playbook -u centos -i devspace_openstack_ip, provision-devspace.yml

Access the devspace and determine ports

Ports to access the various services are dynamically assigned. You will have to log in to the devspace as the omero user to determine the port used by a given service using your usual ssh key and not the id_gh_rsa key:

    ssh omero@devspace_openstack_ip
    cd devspace
  • The port to access the Jenkins UI is obtained by running:

      docker-compose port nginxjenkins 443
    
    • The output of the command looks like:

        WARNING: The JENKINS_PASSWORD variable is not set. Defaulting to a blank string.
        WARNING: The USER_ID variable is not set. Defaulting to a blank string.
        0.0.0.0:xxxx
      

      where xxxx is the assigned port $JENKINS_PORT

    • The Jenkins UI will be available at:

        https://devspace_openstack_ip:$JENKINS_PORT
      
  • The port to access OMERO.web is obtained by running:

     docker-compose port nginx 80 
    
    • The command will generate a similar output that the one above

        WARNING: The JENKINS_PASSWORD variable is not set. Defaulting to a blank string.
        WARNING: The USER_ID variable is not set. Defaulting to a blank string.
        0.0.0.0:xxxx
      

      where xxxx is the assigned port $WEB_PORT

    • OMERO.web will be available at

        http://devspace_openstack_ip:$WEB_PORT/web 
      
  • The port to access OMERO.insight or OMERO.cli is obtained by running:

     docker-compose port omero 4064 
    
    • The command will generate a similar output that the one above

        WARNING: The JENKINS_PASSWORD variable is not set. Defaulting to a blank string.
        WARNING: The USER_ID variable is not set. Defaulting to a blank string.
        0.0.0.0:xxxx
      

      where xxxx is the assigned port $SERVER_PORT

    • To login either via OMERO.insight or OMERO.cli use $SERVER_PORT as the port value and devspace_openstack_ip as the server value. You must use the secure connection.

Job configurations

Job workflow

The default deployment initializes a Jenkins server with a predefined set of jobs.

The table below lists the job names, the Jenkins node labels and the associated docker they are associated with and a short description of the jobs.

Job name Name Description docker name
Trigger Runs all the following jobs in order
BIOFORMATS-push testintegration Merges all Bio-Formats PRs devspace_testintegration_1
BIOFORMATS-build testintegration Builds Bio-Formats components devspace_testintegration_1
BIOFORMATS-image testintegration Builds a Docker image of Bio-Formats devspace_docker_1
OMERO-push testintegration Merges all OMERO PRs devspace_testintegration_1
OMERO-build testintegration Builds OMERO artifacts (server, clients) devspace_testintegration_1
OMERO-server omero Deploys an OMERO.server devspace_omero_1
OMERO-web web Deploys an OMERO.web client devspace_web_1
OMERO-test-integration testintegration Runs the OMERO integration tests devspace_testintegration_1
OMERO-robot testintegration Runs the Robot tests devspace_testintegration_1
nginx nginx Reloads the nginx server devspace_nginx_1
OMERO-docs testintegration Builds the OMERO documentation devspace_testintegration_1

This means that by default the following repositories need to be forked to your GitHub account:

If you do not have some of the repositories forked, you will need to remove the jobs from the list of jobs to run either from the Trigger job configuration or directly from the Jenkins UI i.e. Trigger > Configure.

Default packages used

Name Version Optional
Java openJDK 1.8 openJDK 1.8 devel, oracleJDK 1.8
Python 2.7 -
Ice 3.6 3.5
PostgreSQL 9.4 https://hub.docker.com/_/postgres/
Nginx 1.8 -
Redis latest https://hub.docker.com/_/redis/

Troubleshooting

See Troubleshooting

ADVANCE: extend omero-install

In order to install additional components or new version of packages e.g. PostgreSQL 10, it is required to:

  • Modify the files in omero-install
  • Create a new image of devslave-c7-docker using the updated omero-install files
  • Push the new image to Docker Hub. You will need to your own account
  • Modify each Dockerfile of this repository to use the new image

Upgrade

See Changelog