Skip to content

New test groups #77

Merged
merged 3 commits into from May 7, 2013

2 participants

@joaquincasares

Replaced the integration test group with short and long test groups. I also added a duration test group.

  • short runs unit, and short.
  • long runs unit, short, and long.
  • duration runs all the above + future duration tests (10+ minutes).

I also added a small commit for making the Test logging clearer with the Test Class names.

@pcmanus
DataStax member
pcmanus commented May 6, 2013

"duration" feels like a weird name for a test group, I'd prefer something like "all". Of course, I'm not sure what "future duration tests" are supposed to be.

As a side note, it'll be nice if we had an option to disable the verbose logging of tests (-Dtest.verbose=true or something like that). When you only want to check if all tests passes or not, having your terminal flooded is not necessarily useful. In fact that's a detail but my own preference would go to a non-verbose output by default, with an option to trigger something more verbose (the way nosetests does by default for instance).

@joaquincasares

As a continuation of the discussion in #78, "duration" is a special case that will be run primarily before releases. Duration tests are similar to stress tests, except duration tests run over the course of days rather than many connections over a short amount of time. That being said, the "all" test group would probably work better for the "long" test group since duration is a bit of an edge test group.

Side note: We can do that, but the only issue with that is that the tests become more of a black box since all that is spit out are random errors during CCM reconnections. It then isn't until the end of all the tests that you find out if a test was successful or not. Having run the tests as often, I can tell you it's highly frustrating to have to wait indefinitely for a re-run test to complete.

With these new additions, if there is ever an issue, devs and QA will be able to see the issue as it arises instead of waiting for the entire suite to continue before continuing work.

Also, the defaults for JUnit are to print this type of information, but TestNG decides not to by default. So really, we're just adding an advanced version of what JUnit ships by default.

If you want, we can perhaps add a switch to disable the logging, from the default of on? If so, feel free to add that, or I'll ping Michael, since I was having issues setting that up the first time.

@pcmanus pcmanus merged commit 214b157 into datastax:master May 7, 2013
@pcmanus
DataStax member
pcmanus commented May 7, 2013

duration tests run over the course of days

For info, I'm fine adding that to the testNG tests if that simplify thing in the short term, but longer term, I'm not convinced testNG will be ideal for that kind of testing (you'll probably want to support a wide range of options, record some metrics along the way, and execute this in a more realistic environment that from testNG imo).

It then isn't until the end of all the tests that you find out if a test was successful or not

Pretty sure that doesn't have to be that way. From what I can see of the TestListenerAdapter, it seems easy to have the tests complain right away on errors while being mostly silent for successful ones. Anyway, that's an aside, and I'll have a look at that when I have a minute.

@joaquincasares joaquincasares deleted the joaquincasares:new_test_groups branch May 7, 2013
@joaquincasares

+1 on it not being a long term solution.

-1 on that being the current TestListenerAdapter flow. I'm not sure what you're seeing, but let's discuss this further since output on my end looks like this:

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running TestSuite
Configuring TestNG with: TestNG652Configurator
Starting caseTest [1/63]...
SUCCESS: caseTest
Test: 00:00:00 seconds
Elapsed: 00:00:01 seconds


Starting multiDefinitionTest [2/63]...
SUCCESS: multiDefinitionTest
Test: 00:00:00 seconds
Elapsed: 00:00:01 seconds


Starting DynamicCompositeTypeTest [3/63]...
SUCCESS: DynamicCompositeTypeTest
Test: 00:00:00 seconds
Elapsed: 00:00:06 seconds


Starting collectionTest [4/63]...
SUCCESS: collectionTest
Test: 00:00:01 seconds
Elapsed: 00:02:02 seconds

....

Starting tokenAwarePreparedTest [23/63]...
    730314 [main] INFO  com.datastax.driver.core.Cluster  - New Cassandra host /127.0.1.2 added
    732092 [main] INFO  com.datastax.driver.core.CCMBridge  - Force stopping: 127.0.1.2
    732313 [main] INFO  com.datastax.driver.core.TestUtils  - Waiting for stopped node: 127.0.1.2
    747323 [main] INFO  com.datastax.driver.core.CCMBridge  - Starting: 127.0.1.2
    784060 [Scheduled Tasks-0] INFO  com.datastax.driver.core.Cluster  - Cassandra host /127.0.1.2 removed
    815383 [main] INFO  com.datastax.driver.core.TestUtils  - Waiting for stopped node: 127.0.1.2
SUCCESS: tokenAwarePreparedTest
Test: 00:01:55 seconds
Elapsed: 00:16:24 seconds


Starting tokenAwareTest [24/63]...
    846390 [main] INFO  com.datastax.driver.core.Cluster  - New Cassandra host /127.0.1.2 added
    848113 [main] INFO  com.datastax.driver.core.CCMBridge  - Force stopping: 127.0.1.2
    848337 [main] INFO  com.datastax.driver.core.TestUtils  - Waiting for stopped node: 127.0.1.2
    863343 [main] INFO  com.datastax.driver.core.CCMBridge  - Starting: 127.0.1.2
    874494 [main] INFO  com.datastax.driver.core.CCMBridge  - Error during tests, kept C* logs in /tmp/1367891237919-0
java.lang.AssertionError: For 127.0.1.1 expected [0] but found [1]
    at org.testng.Assert.fail(Assert.java:94)
    at org.testng.Assert.failNotEquals(Assert.java:494)
    at org.testng.Assert.assertEquals(Assert.java:123)
    at org.testng.Assert.assertEquals(Assert.java:370)
    at com.datastax.driver.core.AbstractPoliciesTest.assertQueried(AbstractPoliciesTest.java:90)
    at com.datastax.driver.core.LoadBalancingPolicyTest.tokenAwareTest(LoadBalancingPolicyTest.java:277)
    at com.datastax.driver.core.LoadBalancingPolicyTest.tokenAwareTest(LoadBalancingPolicyTest.java:230)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
    at org.testng.TestRunner.privateRun(TestRunner.java:767)
    at org.testng.TestRunner.run(TestRunner.java:617)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:329)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:291)
    at org.testng.SuiteRunner.run(SuiteRunner.java:240)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
    at org.testng.TestNG.run(TestNG.java:1057)
    at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:77)
    at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeMulti(TestNGDirectoryTestSuite.java:189)
    at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:105)
FAILED: tokenAwareTest
Test: 00:00:34 seconds
Elapsed: 00:16:58 seconds
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.