Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
CLOUDSTACK-10017: Adding 'quick-test' attribute to list of smoketests… #2209
… and adding 'vpc', 'templates', 'deploy-vm', 'roles', 'service-offerings', 'storage', 'routers', 'networks', 'usage', 'accounts' tags
quick-test label aims to provide quick feedback of recent changes to the developer, instead of waiting for the whole smoketest to execute(24h+). This execution lasts about 1.5h at Trillian environments.
The categories aim to offer more efficient way to slice through tests in both smoke and component tests. Let's say as a developer you've made a change in the accounts. You can simply trigger marvin tests with tag=accounts pointing all the tests for example:
Oct 3, 2018
I must share that as a developer I don't see any value in this, as for any change I would do I would run a full smoketest anyway, and for component/resource specific test run I can already run a single test file which are already grouped by resource/component, for example vm_lifecycle test is all related to a vm etc so I could run that individual test.
@borisstoyanov can you pull rebase against latest master?
I've just rebased @rhtyd, a full smoketest would take day+ and this label gives you a real quick feedback if your changes brake anything fundamentally.
Actually in theory smoketests should be really quick and should give near instant result back. Their main outcome should be yes/no answer to 'is the build testable' question (as per common testing practice). I think this is where we got this wrong and kept on adding and adding more items to the suite, instead it should be the other way around, having a really short smoketest and more extensive regression testing suite.
With the current state, if you want to run all tests related to deploying a VM for example you'll have to dig out each individual file which contains these tests. With this extension you'll just need to add label "deploy-vm" to your test execution and it'll check all tests within all the files that have this label. It'll simply save you the hassle of going through each individual file and triage if you need to run it.
in theory i agree @borisstoyanov - the problem there is how do you define fundamental.
We could split the tests in to 3 categories
The broken tests can live in a separate directory, and as they get fixed, can be moved into the known good directory to be run in future.
but by hook or by crook we need to get to a point where a failure really means a code issue.
then we can be much more boolean about quality.