Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC] Improved Subtest-bypass control #125

Closed
cevich opened this issue May 5, 2014 · 3 comments
Closed

[RFC] Improved Subtest-bypass control #125

cevich opened this issue May 5, 2014 · 3 comments

Comments

@cevich
Copy link
Member

cevich commented May 5, 2014

In 0.7.x we have a disabled CSV list that can be more or less hard-coded to force subtests and sub-subtests to raise TestNAError. However this is exploiting faulty logic (this exception is suppose to be a signal for environment or pre-requesite problems). The central issue is we have no access to read from configuration within the control file, and no existing way to pass information from control into subtests.

One possible way we can resolve this, is if the control file is given write-only access to record informational/advisory values into a dedicated config. file. In other words, every time the harness runs, control could write a new config_custom/control.ini. This could then be consulted by subtests/sub-subtests to consider appropriate action.

@cevich cevich added this to the Future Major/Minor Dockertest API revision milestone May 5, 2014
cevich referenced this issue in cevich/autotest-docker May 5, 2014
Enabling tests via ``--args`` on autotest command line
or running all tests is default.  No convenient way
to disable a test from inside dockertest.  The new
``disabled`` option, accepts subtest or subsubtest
CSV to bypass.

Signed-off-by: Chris Evich <cevich@redhat.com>
@cevich
Copy link
Member Author

cevich commented May 5, 2014

dang, in fact, we can't have control write config_custom/control.ini because test-skipping needs to happen via exclusion from the steps engine. So it really does all have to happen inside the control file, and not interact with subtest config at all 😖

@cevich
Copy link
Member Author

cevich commented May 5, 2014

To address #114, one idea might be to use a special character like "~" to invert meaning of --args. i.e. skip the listed tests instead of run only listed tests.

@cevich cevich modified the milestones: Dockertest Version 0.8.1 (API Changes), Future Major/Minor Dockertest API revision May 6, 2014
@cevich
Copy link
Member Author

cevich commented Jun 10, 2014

This issue has been resolved by extending --args processing to include two new tokens: 'i=' and 'x=' to specify specific sub/sub-subtests to include or xclude. This was really difficult to pull off and required extending not only the control file but many aspects of the Subtest, SubSubtest, and SubSubtestCaller classes. One nice side-effect is that it's now possible for SubSubtestCaller's to dynamically generate their own subsubtests at module-load time. All of this is in #109

@cevich cevich closed this as completed Jun 10, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant