Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Don't make test numbers dependent on filters #28

Closed
emichael opened this issue Apr 3, 2022 · 1 comment
Closed

Don't make test numbers dependent on filters #28

emichael opened this issue Apr 3, 2022 · 1 comment
Labels
enhancement New feature or request first issue Good for someone getting started contributing to dslabs

Comments

@emichael
Copy link
Owner

emichael commented Apr 3, 2022

Currently, if you filter out any tests (e.g., by choosing a part of the lab or by only running search tests etc.), all of the tests are renumbered. This is because test numbers are assigned by JUnit after all of the filtering takes place. It would be much nicer to have a consistent numbering scheme. Test numbers should probably have "fully qualified" names, assigned by annotation. For example, the "number" of test 4 in part 2 would be "2.4" (a String rather than an int).

Then, from ./run-tests.py if --part is specified, individual tests can be selected without referring to the fully qualified name (i.e., you can specify --lab 1 --part 2 -n 4 or --lab 1 -n 2.4 but not lab 1 -n 4).

A complication is labs with only one part. Their test numbers shouldn't be 1.1, 1.2 etc. but just 1, 2. And referring to tests this way in ./run-tests.py should be valid.

It would also be nice to sort tests based on the numbering annotation rather than method name. Test numbers could then be removed from method names. I'm not sure if this is possible with the version of JUnit we're using.

Once question is whether the annotation should have the fully qualified name, or a simple integer and pick up the part number from the test class. There are arguments for both approaches, but having the fully qualified name (e.g., @TestNumber("2.4")) is probably best because it allows students to easily look at the method and know how to run it.

Lastly, it would be great if we could validate in an automated test that test numbers are sequential, there are no duplicates, every test has a number, etc. This should go in the tst-self directory.
https://github.com/emichael/dslabs/tree/master/framework/tst-self/
@emichael emichael added enhancement New feature or request first issue Good for someone getting started contributing to dslabs labels Apr 3, 2022
emichael added a commit that referenced this issue Apr 17, 2022
@emichael
Giving tests unique numbers based on part. Fixes #28.
53fb41a 
Previously, when tests where filtered out the numbers of other tests
were recalculated. This commit assigns unique test numbers based on the
@part annotation (if present) and the test name itself.

This commit also adds the @Lab annotation and looks up tests suites and
viz configs based on their annotations by searching through the
classpath. This obviates the need for the uniquely-named classes in the
testsuites and vizconfigs packages.
@emichael
Copy link
Owner Author

Fixed in 53fb41a

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request first issue Good for someone getting started contributing to dslabs
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@emichael