Add a proper development VM setup

[ondrejbudai]

I'm tired of me and other people not knowing how to properly run osbuild-composer when developing it. A lot of people don't like running code-under-development on their local machine especially when parts of it are running with superuser privileges. To solve this issue I took Martin's Vagrant PR (#234) and turned it into a proper development VM setup. I consider this PR as good enough to be merged but it's mostly just MVP - it can be expanded by a lot.

Quick example

Run:

make vm-provision vm-install vm-test

This command will provision the VM, build and install osbuild-composer and run all the integration tests.

How it works

I added three targets to Makefile which enable a user to provision the VM, install osbuild-composer from source to it and finally run the integration tests. I tried to document the process as much as possible in the README.

Technical details

Osbuild is installed from the upstream master branch. The RPM is built locally using make rpm. Osbuild-composer is installed from the HEAD of local checkout. The RPM is also built locally using make rpm.

Notes

• building osbuild and osbuild-composer rpms in koji would be nice. The downside is that is slower and requires additional setup (kerberos login), this might be bad for open source project. There's also an option to build the rpms locally with mock.
• testing multiple fedora versions. Switch to mock/koji to build the rpms is must have here. Also, I'm not sure how to deal with it using Vagrant, we might want to switch to use plain qemu for it.
• testing RHEL. Switch to mock/brew is required. Vagrant boxes might be problem as well, we can use plain qemu here as well.
• testing multiarch. Imho not possible/viable.
• why Vagrant was used if it's not flexible enough to deal with multiple Fedora versions and RHEL?
  Because it makes things easier - by default it can transfer files to the VM using rsync and provides a simple way to log in it using ssh. Also it very easily manages port redirects and state (downloading the box and spinning up the VM). We can switch to something more flexible later but it will require more setup.
• how does this relate to the current CI effort?
  The CI effort uses ansible to provision the VM, we can reuse the playbooks with some modifications to provision the development VM. We can also reuse the scripts to build the RPMs. In my opinion this is enough to create environment as close to the "official CI environment" as possible.
• osbuild from master?
  As discussed earlier we should stick to some version of osbuild. @teg recently did some work in building osbuild from specific commit. Once the required parts are merged, it should be possible to use it in this PR.
• multi-machine integration tests. These are required for multiarch, which is imho not possible/viable as stated previously. The only other thing that requires multiple machines is the remote worker. This might be tricky but it's definitely doable.

[msehnout]

testing multiple fedora versions. Switch to mock/koji to build the rpms is must have here. Also, I'm not sure how to deal with it using Vagrant, we might want to switch to use plain qemu for it.

We can run make rpm directly inside the Vagrant box. This way it won't be necessary to use mock. See my development setup (works also on mac):
https://github.com/msehnout/osbuild-dev-vm/

testing RHEL. Switch to mock/brew is required. Vagrant boxes might be problem as well, we can use plain qemu here as well.

Again not necessary, if we build the RPM in the Vagrant box and if we can get #227 done :-)

multi-machine integration tests. These are required for multiarch, which is imho not possible/viable as stated previously. The only other thing that requires multiple machines is the remote worker. This might be tricky but it's definitely doable.

Multi machine setup with Vagrant should be easy. Multi-arch might be tricky, because you would need different "providers" for each VM and of course it is a pain to use software virtualization ...

Otherwise I like this PR very much :-), but to be clear: what is the goal of this PR? Do we want to run integration tests in the VM, develop osbuild using it, or both? Because PR #234 was only about testing, I proposed https://github.com/msehnout/osbuild-dev-vm/ for development.

[ondrejbudai]

We can run make rpm directly inside the Vagrant box. This way it won't be necessary to use mock. See my development setup (works also on mac):
https://github.com/msehnout/osbuild-dev-vm/

My thought is that we don't want to do anything extra in the VM to make it as pristine as possible.

Multi machine setup with Vagrant should be easy. Multi-arch might be tricky, because you would need different "providers" for each VM and of course it is a pain to use software virtualization ...

That's right, I just don't have any experience with it and the work is needed to be done do distribute tls key-pairs, so I just consider it somewhat tricky.

Otherwise I like this PR very much :-), but to be clear: what is the goal of this PR? Do we want to run integration tests in the VM, develop osbuild using it, or both? Because PR #234 was only about testing, I proposed https://github.com/msehnout/osbuild-dev-vm/ for development.

Both. I think it's a good idea to keep the VM as pristine as possible, therefore I think it's OK to use it for both development and integration testing. If you want to be 100% sure to have it pristine before running the integration tests, you can always drop and reprovision it.

[teg]

Thanks for working on this @ondrejbudai and @msehnout. I love the overall goal, but have some concerns about the details.

Building RPMs

rpmbuild / spec files has all the problems we are trying to fix with osbuild: not reproducible and dependent on their environment. With that in mind, I think the only way to build rpms is "very carefully". That is, we need a build environment that is as predictable as possible and as close as possible to the one that will be used to build the official RPMs. For local development I think we should use mock and whenever feasible I think we should use scratch builds in koji directly.

Building in the VM we will be testing in means pulling in build dependencies which would otherwise not be pulled in by the test rpms, which I think we should avoid (and also risks the state of the system not being pristine, and affecting the build). I think we should also build rpms separately from the test images for the purposes of reproducibility.

Multiple Fedora versions

I think we should not make any assumptions about the host Fedora version. I think mock deals with this just fine, and in the case people develop on non-Fedora machines, I think koji would be a tolerable requirement.

Packit-like functionality

Some of the above problems would be solved once we have RPMs built per-PR either by packit or by our own service (maybe we need to start off with our own, but let's get back to that separately).

My ideal work-flow

I think we need to first set up and pin a well-known base VM (up-to-date Fedora Cloud 31 without any extra packages installed). We should take snapshots of this to run our tests in, knowing that every time we run the test-suite the base image is bit-for-bit the same and the only thing that changed is our test RPMs. From time to time we can update the base VM, but that should then be done explicitly and the last known-good tests re-run to verify the changing image does not break them. In CI the base VM should be pinned globally, but on each dev machine, I think it is still fine (though not ideal) that each developer manages this manually. What we should avoid is to automatically update the base VM with every test as we will then lose control over what changed.

Doing a test-run should, in my view, just instantiate and boot the base VM, copy in and install the test rpms, start composer (though we could discuss if the tests should do this) and then call each test binary one-by-one. Note that no other setup is done, no other files are copied in and no other packages installed. The idea is that everything that can possibly go wrong should be either part of the base image or the rpms, so in particular, I never want to deal with subtle problems stemming from the version of the test-infrastructure being used.

The script I use:

#!/bin/bash
scp rpms/*.rpm test-vm:~
ssh test-vm sudo dnf localinstall "*.rpm"
ssh test-vm sudo systemctl start osbuild-composer
ssh test-vm sudo /usr/libexec/tests/osbuild-composer/osbuild-tests
ssh test-vm sudo /usr/libexec/tests/osbuild-composer/osbuild-dnf-json-tests

Vagrant

I'm not yet familiar enough with all the features of Vagrant to be confident in making it a requirement for doing development (which I think the ability of running integration tests locally should be), so I'll just leave the properties I think we want in the section above and then we can discuss Vagrant vs qemu (which would be my default preference as I understand how it works).

[msehnout]

I agree with you @teg , though I think we are discussing 2 different things here. What are our expectations from this: Do we want a development VM or a clean VM which we will only use for testing? I want both, but now I think these are 2 different setups.

Regarding Vagrant:
Although I like to use plain qemu for simple tasks I really enjoy the convenience of using Vagrant. It is basically a wrapper for libvirt, which is in turn API for qemu. My wish is to use 3 VMs in the Vagrantfile. One for osbuild-composer, one for the worker, and one for Koji. We might end up spinning another one for a client. In this scenario the Vagrant will set up a virtual network for us and make the whole process a lot easier than plain qemu would.

[ondrejbudai]

I discussed this PR with @teg and created #255 which sums up our discussion. I will close this PR for now because the discussed approach differs a bit. However, feel free to use this PR, it works just alright.