Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Test runner improvements #21

jakub-g opened this Issue Jun 28, 2012 · 2 comments


None yet
1 participant

jakub-g commented Jun 28, 2012

This is to suggest several small improvements to our test runner, so manual testing at the end of each sprint is less painful.

My proposals:

  1. When a test suite is finished, it should be very clearly indicated, e.g. by changing the background color of the page. That way we can easily see whether the suite is finished, or it hanged on some of the tests. Looking for the status text requires more attention.

  2. We should extract the test cases that

  • are somehow 'problematic'
  • take a long time,
  • require focus,
    and move them to be launched at the end of the suite (80/20 rule). It's easier to manually skip the tests at the end of the suite, than to check every 20 seconds if the suite is running.
  1. Perhaps we should add a query string option, that instead of running the whole suite, would list all the tests (not recursive, only the top-level with respect to the classpath) as a clickable URLs?

Something like http://aria/.../test/?autostart=false

As we know, it's not advisable due to some reasons to launch the whole test suite from the root (especially, when it breaks in the middle, you don't know where to restart). Current way of doing it is, from what I see, to open a test suite JS file, and copy the classpaths of the tests from sub-suites (tedious).

For now, I've listed the classpaths as links in our internal Aria_Templates_Delivery_Guide, but this has a drawback that it must be kept in sync with the JS file, and we all know at some stage someone will forget to update both at the same time. It's safer to add a parameter to the test runner.

  1. (partially overlapping 3) Basically, I think it would be good to have a pre-index page in the test runner (parametrized by ?classpath=...), on which you will have the links to
    -- start the mentioned suite
    -- to list the tests in the mentioned suite
    -- additionaly, somehow choose whether the previous should be done with "runIsolated=true" or not.

Go on with your thoughts on this, and suggest your improvements! Feedback very welcome.


jakub-g commented Sep 14, 2012

  1. is solved with the ?robot=false param.

The other parts - to be thought of. Perhaps if we move to the new testing solution and it works as we want, this issue will be WONTFIX.


jakub-g commented Mar 22, 2013

Most of the problems described here are no longer relevant due to migration to Attester. Closing as WONTFIX.

@jakub-g jakub-g closed this Mar 22, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment