Create basic acceptance tests for service-manager

[hferentschik]

No description provided.

[hferentschik]

Relates to issue #194

[optak]

not sure what you mean by acceptance test, but we have sanity test for service-manager env here:
https://github.com/projectatomic/adb-tests/blob/master/testsuite/Components/vagrant/service-manager/sanity/runtest.sh

[hferentschik]

not sure what you mean by acceptance test, but we have sanity test for service-manager env here

Right, something along these lines, but using vagrant-spec, aka a Ruby testing framework. The plugin is written in Ruby and it makes sense to stay in this idiom. Also one has all the universe of Ruby to write tests. Also the tests would be living with the code, so the developer can use them during development (actually being able to debug the code), but also to verify that all tests are still passing when creating a pull request. In a perfect world, we then even have a CI server which picks up the pull request and verifies it against the test. @optak, on that note, do we have infrastructure for a CI server?

BTW, it will look very similar to this - projectatomic/adb-vagrant-registration@81ed5c6

The idea is to be able to run the tests out of the box against VirtualBox and Libvirt.

[optak]

Thanks for info and example @hferentschik. So you are talking about unit tests? In CDK QE we focus more on integration and testing on different environment, so it could be:

1. unit tests run upon git commit and written in the same language as tested code
2. end2end scenarios, integration testing: use whatever framework and language (bash + beakerlib is not final, it can change), living in https://github.com/projectatomic/adb-tests, in future with some CI infrastructure as well.
   Running all tests from that repo takes probably too much time for testing each commit anyway.

[hferentschik]

So you are talking about unit tests?

Unit tests as well, yes. They would most likely use be using straight RSpec tests. vagrant-specs are really acceptance tests. Or call it black box tests if you like. They make vagrant plugin calls expecting certain outcome from it w/o knowing how or what the plugin does. Are these acceptance tests as per QA. I guess they could and IMO they could be reused, but that's a different discussion.

In CDK QE we focus more on integration

the vagrant-spec tests will be integration tests

and testing on different environment

That part is imo most critical. That's not something covered directly by these tests. Obviously these test should run on various environment (the one the dev works on), but they are not trying to somehow automatically run on various platform (eg via virtualization or so).

1. unit tests run upon git commit and written in the same language as tested code

this are more than unit tests we are talking about. Let's just say, these tests are run during development and on basic CI (something which builds for example pull requests). How you exactly classify/label these tests might depend on your point of view.

2. end2end scenarios, integration testing: use whatever framework and language (bash + beakerlib is not final, it can change), living in https://github.com/projectatomic/adb-tests, in future with some CI infrastructure as well.

I put it more like this - QE is responsible to make sure that the software covers all requirements. This might include running tests on various OSes or different versions of the VM, etc. In my view, QE should reuse as much of the existing tests and potentially also contribute tests back into main repository. Any kind of test which does not fit into the main repository and belongs more to an overall adb testing strategy, adb-tests is used. Most likely there will be some overlap, but I would try to minimize that. Also adb-tests would indeed have its own CI infrastructure.

Running all tests from that repo takes probably too much time for testing each commit anyway.

I assume you mean "running all test from adb-utils on each commit takes to much time" - probably.

The idea is really to give the developer some tools to be able to refactor code and/or add new behavior w/o breaking existing functionality and requirements. To do so, the developer needs a test suite he can run directly from the code he is working on. How you call these tests does not matter so much. Most likely it will be a mixture of vagrant-spec tests which are quite high level, but are unfortunately slow, since they spin up the VM. I fully expect that there will be pure unit tests which will just test single classes. This, however, will itself require some refactoring to make parts of the code more testable (a side goal of mine would be to be able to be able to mock certain VM behavior in some tests to avoid bootstrapping the VM altogether). To do so, it will be good to have the high level tests first to make sure that the refactoring won't break the overall behavior.

@optak does that make sense

[hferentschik]

I created #196 in order to cover the aspect that a plain unit test framework is needed as well.