Work out a CI setup

[ncoghlan]

Since we have integration tests now, we have the ability to set up some pre-merge CI. However, the integration tests currently rely on Vagrant with 64-bit CentOS boxes which means either:

• figuring out a way to switch them to a 32-bit VM that actually works in Travis (as per http://stackoverflow.com/questions/37307114/launching-a-vagrant-vm-inside-travis-ci ); or
• talking to Karanbir et al about setting something up in the CentOS CI infrastructure instead

@shaded-enmity would you mind talking to @msrb about that? He's been doing a bunch of Jenkins/GitHub integration lately, and I think that might actually be our best option here.

[shaded-enmity]

Yup, as a matter of fact we talked about this exactly yesterday :) After drawing bunch of diagrams on the table I realized that we might be dead in the water, since we need to interface w/ libvirt, we really need to either:

• Execute the tests on bare metal
• Execute the tests in a nested virtualization enabled VM

@kbsingh do you now if either of the two options would be applicable on CentOS CI?

[shaded-enmity]

(I have a backup plan of setting up a Jenkins on cucos-1/2 with nested virt enabled)

[kbsingh]

Running bare metal is no problem on @centos-ci - All you would need to do is make sure there is a script included that does the machine setup before running the tests ( eg. yum install libvirtd && service libvirtd start etc ). This works as long as the test runtime is under a few hours, if you need to run the tests in a way that each individual run would take more than a few hrs, then @centos-ci has VMs as well, which all have nested virt enabled. But I would recommend going down the route o bare metal first.

[kbsingh]

to make life easier, if you need Vagrant to support the tests you are running right now, you can always just install vagrant from SCLs and just use that as well :)

The typical machine size on @centos-ci is 8cores 32gb ram E3-1270; so you should have enough capacity based on my initial scan of the code right now.

There is a Getting Started walk through at https://wiki.centos.org/QaWiki/CI/GettingStarted - Let me know if you need anymore info.

[shaded-enmity]

Thanks KB, following the walk through now. Will add bug tracking number later.

[ncoghlan]

@shaded-enmity Something that the CI is going to need that isn't strictly necessary when running tests locally is passwordless sudo access for the leapp-tool.py command.

If the tests run as root, that would solve that problem, otherwise we may need to drop a config file into /etc/sudoers.d.

[shaded-enmity]

@ncoghlan good point, thanks, will prepare Ansible playbook for setting up the test machine & environment once I get my hands on it.

[kbsingh]

Note that centosCI will give you root@ the worker nodes by default

[shaded-enmity]

Something to consider now is the caching of Vagrant boxes, every test run requires centos:6 and centos:7 box images to be present on the system executing the tests. Pulling the images, depending on network quality, might actually take more time, or at least be a major component of, than running the test themselves.

[ncoghlan]

Good point, my initial setup was based on the assumption of a persistent workstation, and that's not valid in a CI environment.

@kbsingh Is there anything already set up in the CentOS CI environment that could help with this?

Note that while we'd definitely prefer it, "Vagrant box" isn't actually a hard requirement - if there were non-Vagrant libvirt compatible images available, we could switch over to using Ansible to configure the test VMs instead.

Vagrant is just handy since it automatically deals with the VM network and storage configuration for us.

[bstinsonmhk]

WRT caching the vagrant boxes, we should already be optimized. Atlas redirects the download of the box itself to cloud.centos.org, and we have a mirror of that local to the CentOS CI Environment

[ncoghlan]

@bstinsonmhk Nice! It's always a happy outcome when it turns out the convenient approach has already been made efficient :)

[ncoghlan]

@shaded-enmity Looking up something else in the behave docs, I realised that since we're going to be using Jenkins, we should run behave with the --junit option: https://pythonhosted.org/behave/behave.html#cmdoption--junit

[ncoghlan]

And another potentially relevant config option: https://pythonhosted.org/behave/tutorial.html#controlling-things-with-tags

However, to simplify test skipping, I think it will make sense for us to adopt the autoskipping approach recommended in http://stackoverflow.com/questions/36482419/how-do-i-skip-a-test-in-the-behave-python-bdd-framework/42721605#42721605

That's actually useful to me for #29 to mark the depends-on-the-DBus-service-I-haven't-written-yet scenario for skipping, so I'll add it there.

[bstinsonmhk]

The workspace and other accounts are created on CICO, are you folks working on a JJB template for your jobs?

[ncoghlan]

Our CentOS pre-merge CI has been working for a while now :)