Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a only-tags option for test_all #459

Merged
merged 1 commit into from Jan 14, 2017

Conversation

commial
Copy link
Contributor

@commial commial commented Dec 22, 2016

Add the possibility to launch tests regarding only a subset of tag (whitelist version of omit-tags).

For instance:

$ python test_all.py -o z3
...
Result: 27/27 pass

This is quite useful while, for instance, refactoring a given Miasm segment...

@@ -703,15 +714,15 @@ class ExampleJitterNoPython(ExampleJitter):
"'py-llvm 3.2' module is required for llvm tests"

# Remove llvm tests
if TAGS["llvm"] not in exclude_tags:
if (not include_tags and TAGS["llvm"] not in exclude_tags):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe there is an unmanaged case where you only include a tag which will pull a dependency on tcc for example, on a machine which does not support tcc.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How should it be handled? Drop the test case? Silently? Abort?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh! I don't know: I was not sure about me on the review of this case.
I may prefer the abort case, but it's up to you.
If you are aware of the case and prefer to let the code like this, tell me.

@commial
Copy link
Contributor Author

commial commented Dec 22, 2016

So now, there is 2 distincts part:

  1. Creation of the testset (full, or tags mark as included with their dependencies)
  2. Pruning of the testset according to the exclusion list

To avoid unwanted behavior, the impossibility of using both inclusion and exclusion is conserved in the CLI handling.
But now, if I ask for, say, llvm,tcc, and llvm is not supported on my install, I end with tcc tests and llvm tests dependencies.

It seems fine for me, do you agree?

@commial
Copy link
Contributor Author

commial commented Jan 14, 2017

Finally, I removed this weird comportement, which seems unexpected.

Now, there are 2 options:

  • -t OMIT_TAGS: blacklist mode
  • -o ONLY_TAGS: whitelist mode

They can be used together.
For instance, -o tcc -t example,qemu will only launch regression tests (without QEMU) on TCC and their corresponding dependencies.

@serpilliere serpilliere merged commit 6fef06f into cea-sec:master Jan 14, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants