New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CLOUDSTACK-10017: Adding 'quick-test' attribute to list of smoketests… #2209

Open
wants to merge 2 commits into
base: master
from

Conversation

Projects
None yet
4 participants
@borisstoyanov
Copy link
Contributor

borisstoyanov commented Jul 28, 2017

… and adding 'vpc', 'templates', 'deploy-vm', 'roles', 'service-offerings', 'storage', 'routers', 'networks', 'usage', 'accounts' tags

quick-test label aims to provide quick feedback of recent changes to the developer, instead of waiting for the whole smoketest to execute(24h+). This execution lasts about 1.5h at Trillian environments.

The categories aim to offer more efficient way to slice through tests in both smoke and component tests. Let's say as a developer you've made a change in the accounts. You can simply trigger marvin tests with tag=accounts pointing all the tests for example:


nosetests --with-marvin --marvin-config=[config] --hypervisor=xenserver -a tags=accounts /tests/integration/*/*.*

https://cwiki.apache.org/confluence/display/CLOUDSTACK/Test+Categories+and+Quick-test+in+Marvin

@rhtyd

This comment has been minimized.

Copy link
Member

rhtyd commented Aug 2, 2017

Good effort, @borisstoyanov can you rebase Github is such a complainer :)

@borisstoyanov borisstoyanov force-pushed the borisstoyanov:add-quick-test-attribute branch from 02a715e to 8297e60 Aug 2, 2017

@borisstoyanov

This comment has been minimized.

Copy link
Contributor

borisstoyanov commented Aug 2, 2017

Thanks @rhtyd, I've rebased and squashed all the commits

@borisstoyanov

This comment has been minimized.

Copy link
Contributor

borisstoyanov commented Sep 8, 2017

@borisstoyanov borisstoyanov force-pushed the borisstoyanov:add-quick-test-attribute branch from 731e808 to 96cf77b Oct 3, 2018

@rafaelweingartner rafaelweingartner force-pushed the apache:master branch from 1fa4f10 to fcc87d9 Oct 7, 2018

@borisstoyanov

This comment has been minimized.

Copy link
Contributor

borisstoyanov commented Oct 11, 2018

@rhtyd

This comment has been minimized.

Copy link
Member

rhtyd commented Oct 11, 2018

I must share that as a developer I don't see any value in this, as for any change I would do I would run a full smoketest anyway, and for component/resource specific test run I can already run a single test file which are already grouped by resource/component, for example vm_lifecycle test is all related to a vm etc so I could run that individual test.

@borisstoyanov can you pull rebase against latest master?

borisstoyanov added some commits Jul 27, 2017

CLOUDSTACK-10017: Adding 'quick-test' attribute to list of smoketests…
… and adding 'vpc', 'templates', 'deploy-vm', 'roles', 'service-offerings', 'storage', 'routers', 'networks', 'usage' tags
CLOUDSTACK-10017: Adding 'quick-test' attribute to list of smoketests…
… and adding 'vpc', 'templates', 'deploy-vm', 'roles', 'service-offerings', 'storage', 'routers', 'networks', 'usage' tags

@borisstoyanov borisstoyanov force-pushed the borisstoyanov:add-quick-test-attribute branch from 96cf77b to c699b53 Oct 11, 2018

@borisstoyanov

This comment has been minimized.

Copy link
Contributor

borisstoyanov commented Oct 11, 2018

I've just rebased @rhtyd, a full smoketest would take day+ and this label gives you a real quick feedback if your changes brake anything fundamentally.

Actually in theory smoketests should be really quick and should give near instant result back. Their main outcome should be yes/no answer to 'is the build testable' question (as per common testing practice). I think this is where we got this wrong and kept on adding and adding more items to the suite, instead it should be the other way around, having a really short smoketest and more extensive regression testing suite.

With the current state, if you want to run all tests related to deploying a VM for example you'll have to dig out each individual file which contains these tests. With this extension you'll just need to add label "deploy-vm" to your test execution and it'll check all tests within all the files that have this label. It'll simply save you the hassle of going through each individual file and triage if you need to run it.

@PaulAngus

This comment has been minimized.

Copy link
Contributor

PaulAngus commented Oct 11, 2018

in theory i agree @borisstoyanov - the problem there is how do you define fundamental.
and given how large the component suite is (and how badly broken), results from component tests tend to get ignored.

We could split the tests in to 3 categories

  • a set of true smoke tests basic vm lifecycle and network tests that take just a hour or two to run
  • a set of known good component tests - the remainder of smoke tests and known good component tests
  • broken tests - ones that rarely, if ever pass

The broken tests can live in a separate directory, and as they get fixed, can be moved into the known good directory to be run in future.

but by hook or by crook we need to get to a point where a failure really means a code issue.

then we can be much more boolean about quality.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment