Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
New test groups #77
Replaced the integration test group with short and long test groups. I also added a duration test group.
I also added a small commit for making the Test logging clearer with the Test Class names.
"duration" feels like a weird name for a test group, I'd prefer something like "all". Of course, I'm not sure what "future duration tests" are supposed to be.
As a side note, it'll be nice if we had an option to disable the verbose logging of tests (-Dtest.verbose=true or something like that). When you only want to check if all tests passes or not, having your terminal flooded is not necessarily useful. In fact that's a detail but my own preference would go to a non-verbose output by default, with an option to trigger something more verbose (the way nosetests does by default for instance).
As a continuation of the discussion in #78, "duration" is a special case that will be run primarily before releases. Duration tests are similar to stress tests, except duration tests run over the course of days rather than many connections over a short amount of time. That being said, the "all" test group would probably work better for the "long" test group since duration is a bit of an edge test group.
Side note: We can do that, but the only issue with that is that the tests become more of a black box since all that is spit out are random errors during CCM reconnections. It then isn't until the end of all the tests that you find out if a test was successful or not. Having run the tests as often, I can tell you it's highly frustrating to have to wait indefinitely for a re-run test to complete.
With these new additions, if there is ever an issue, devs and QA will be able to see the issue as it arises instead of waiting for the entire suite to continue before continuing work.
Also, the defaults for JUnit are to print this type of information, but TestNG decides not to by default. So really, we're just adding an advanced version of what JUnit ships by default.
If you want, we can perhaps add a switch to disable the logging, from the default of on? If so, feel free to add that, or I'll ping Michael, since I was having issues setting that up the first time.
added a commit
this pull request
May 7, 2013
For info, I'm fine adding that to the testNG tests if that simplify thing in the short term, but longer term, I'm not convinced testNG will be ideal for that kind of testing (you'll probably want to support a wide range of options, record some metrics along the way, and execute this in a more realistic environment that from testNG imo).
Pretty sure that doesn't have to be that way. From what I can see of the TestListenerAdapter, it seems easy to have the tests complain right away on errors while being mostly silent for successful ones. Anyway, that's an aside, and I'll have a look at that when I have a minute.
+1 on it not being a long term solution.
-1 on that being the current TestListenerAdapter flow. I'm not sure what you're seeing, but let's discuss this further since output on my end looks like this: