Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow running more than one WML unit test per instance of Wesnoth. #4535

Open
Pentarctagon opened this issue Nov 1, 2019 · 2 comments

Comments

@Pentarctagon
Copy link
Member

@Pentarctagon Pentarctagon commented Nov 1, 2019

Currently on Travis, the WML unit tests can take a significant amount of time to complete since they each individual unit test starts up its own instance of Wesnoth to run that single test. Given that there are a decent number of tests, it can end up taking 10+ minutes for them to complete, which has caused travis timeouts occasionally.

It is done this way, since currently the WML tests script uses Wesnoth's return code to determine if the test returned the expected value.

The change would then be to instead allow specifying multiple tests to run, and then outputting the results of each test after the final test runs.

@stevecotton

This comment has been minimized.

Copy link
Contributor

@stevecotton stevecotton commented Nov 1, 2019

Of those tests, 160 expect return code zero, leaving 33 that expect non-zero. Supporting running multiple tests only for the ones that are expected to return zero is 80% of the optimization, and probably simpler than handling the non-zero return codes.

@Pentarctagon

This comment has been minimized.

Copy link
Member Author

@Pentarctagon Pentarctagon commented Nov 1, 2019

I think probably the only required C++ change needed would be to add support for running multiple tests in a row - maybe just a huge list at the command line or read from a config file - and then the existing unit tests could be updated to instead log their output to stdout, use PersistenceWML, etc.

The results could also be directly output into a csv file, I suppose.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.