Junit XML reporter #5
Comments
|
First cut of this has been committed, but it's not fully tested yet and more examples of real JUnit output would be useful |
|
Jenkins does not like the current output: Here is an exampe of a valid junit output as generated by nose: <?xml version="1.0" encoding="UTF-8"?>
<testsuite name="nosetests" tests="175" errors="0" failures="0" skip="0">
<testcase classname="pyrad.tests.testBidict.BiDictTests" name="testBackwardAccess" time="0"/>
<testcase classname="pyrad.tests.testBidict.BiDictTests" name="testBackwardDeletion" time="0"/>
<testcase classname="pyrad.tests.testBidict.BiDictTests" name="testDeletion" time="0"/>
<testcase classname="pyrad.tests.testBidict.BiDictTests" name="testForwardAccess" time="0"/>
<testcase classname="pyrad.tests.testBidict.BiDictTests" name="testItemAccessor" time="0"/>
<testcase classname="pyrad.tests.testBidict.BiDictTests" name="testLength" time="0"/>
<testcase classname="pyrad.tests.testBidict.BiDictTests" name="testStartEmpty" time="0"/>
<testcase classname="pyrad.tests.testClient.ConstructionTests" name="testNamedParameters" time="0"/>
<testcase classname="pyrad.tests.testClient.ConstructionTests" name="testParameterOrder" time="0"/>
<testcase classname="pyrad.tests.testClient.ConstructionTests" name="testSimpleConstruction" time="0"/>
<testcase classname="pyrad.tests.testClient.OtherTests" name="testCreateAcctPacket" time="0"/>
<testcase classname="pyrad.tests.testClient.OtherTests" name="testCreateAuthPacket" time="0"/>
<testcase classname="pyrad.tests.testClient.SocketTests" name="testAuthDelay" time="2"/>
<testcase classname="pyrad.tests.testClient.SocketTests" name="testBind" time="0"/>
<testcase classname="pyrad.tests.testClient.SocketTests" name="testBindClosesSocket" time="0"/>
<testcase classname="pyrad.tests.testClient.SocketTests" name="testDoubleAccountDelay" time="3"/>
<testcase classname="pyrad.tests.testClient.SocketTests" name="testDoubleRetry" time="0"/>
<testcase classname="pyrad.tests.testClient.SocketTests" name="testIgnorePacketError" time="1"/>
<testcase classname="pyrad.tests.testClient.SocketTests" name="testInvalidReply" time="1"/>
<testcase classname="pyrad.tests.testClient.SocketTests" name="testNoRetries" time="0"/>
<testcase classname="pyrad.tests.testClient.SocketTests" name="testReopen" time="0"/>
<testcase classname="pyrad.tests.testClient.SocketTests" name="testSendPacket" time="0"/>
<testcase classname="pyrad.tests.testClient.SocketTests" name="testSingleAccountDelay" time="2"/>
<testcase classname="pyrad.tests.testClient.SocketTests" name="testSingleRetry" time="0"/>
<testcase classname="pyrad.tests.testClient.SocketTests" name="testValidReply" time="0"/>
<testcase classname="pyrad.tests.testDictionary.AttributeTests" name="testConstructionParameters" time="0"/>
<testcase classname="pyrad.tests.testDictionary.AttributeTests" name="testInvalidDataType" time="0"/>
<testcase classname="pyrad.tests.testDictionary.AttributeTests" name="testNamedConstructionParameters" time="0"/>
<testcase classname="pyrad.tests.testDictionary.AttributeTests" name="testValues" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryInterfaceTests" name="testContainment" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryInterfaceTests" name="testEmptyDictionary" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryInterfaceTests" name="testReadonlyContainer" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryParsingTests" name="testAttributeEncryptionError" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryParsingTests" name="testAttributeOptions" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryParsingTests" name="testAttributeTooFewColumnsError" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryParsingTests" name="testAttributeUnknownTypeError" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryParsingTests" name="testAttributeUnknownVendorError" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryParsingTests" name="testBeginVendorParsing" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryParsingTests" name="testBeginVendorTooFewColumns" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryParsingTests" name="testBeginVendorUnknownVendor" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryParsingTests" name="testDictFileParseError" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryParsingTests" name="testDictFilePostParse" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryParsingTests" name="testEndVendorParsing" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryParsingTests" name="testEndVendorUnbalanced" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryParsingTests" name="testEndVendorUnknownVendor" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryParsingTests" name="testInclude" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryParsingTests" name="testIntegerValueParsing" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryParsingTests" name="testParseEmptyDictionary" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryParsingTests" name="testParseMultipleDictionaries" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryParsingTests" name="testParseSimpleDictionary" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryParsingTests" name="testStringValueParsing" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryParsingTests" name="testValueForUnknownAttributeError" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryParsingTests" name="testValueTooFewColumnsError" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryParsingTests" name="testVenderTooFewColumnsError" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryParsingTests" name="testVendorFormatError" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryParsingTests" name="testVendorFormatSyntaxError" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryParsingTests" name="testVendorOptionError" time="0"/>
<testcase classname="pyrad.tests.testDictionary.DictionaryParsingTests" name="testVendorParsing" time="0"/>
<testcase classname="pyrad.tests.testHost.ConstructionTests" name="testNamedParameters" time="0"/>
<testcase classname="pyrad.tests.testHost.ConstructionTests" name="testParameterOrder" time="0"/>
<testcase classname="pyrad.tests.testHost.ConstructionTests" name="testSimpleConstruction" time="0"/>
<testcase classname="pyrad.tests.testHost.PacketCreationTests" name="testCreateAcctPacket" time="0"/>
<testcase classname="pyrad.tests.testHost.PacketCreationTests" name="testCreateAuthPacket" time="0"/>
<testcase classname="pyrad.tests.testHost.PacketCreationTests" name="testCreatePacket" time="0"/>
<testcase classname="pyrad.tests.testHost.PacketSendTest" name="testSendPacket" time="0"/>
<testcase classname="pyrad.tests.testHost.PacketSendTest" name="testSendReplyPacket" time="0"/>
<testcase classname="pyrad.tests.testPacket.AcctPacketConstructionTests" name="testBasicConstructor" time="0"/>
<testcase classname="pyrad.tests.testPacket.AcctPacketConstructionTests" name="testConstructWithDictionary" time="0"/>
<testcase classname="pyrad.tests.testPacket.AcctPacketConstructionTests" name="testConstructorDefaults" time="0"/>
<testcase classname="pyrad.tests.testPacket.AcctPacketConstructionTests" name="testConstructorIgnoredParameters" time="0"/>
<testcase classname="pyrad.tests.testPacket.AcctPacketConstructionTests" name="testConstructorRawPacket" time="0"/>
<testcase classname="pyrad.tests.testPacket.AcctPacketConstructionTests" name="testConstructorWithAttributes" time="0"/>
<testcase classname="pyrad.tests.testPacket.AcctPacketConstructionTests" name="testNamedConstructor" time="0"/>
<testcase classname="pyrad.tests.testPacket.AcctPacketTests" name="testCreateReply" time="0"/>
<testcase classname="pyrad.tests.testPacket.AcctPacketTests" name="testRequestPacket" time="0"/>
<testcase classname="pyrad.tests.testPacket.AcctPacketTests" name="testRequestPacketSetsId" time="0"/>
<testcase classname="pyrad.tests.testPacket.AcctPacketTests" name="testVerifyAcctRequest" time="0"/>
<testcase classname="pyrad.tests.testPacket.AuthPacketConstructionTests" name="testBasicConstructor" time="0"/>
<testcase classname="pyrad.tests.testPacket.AuthPacketConstructionTests" name="testConstructWithDictionary" time="0"/>
<testcase classname="pyrad.tests.testPacket.AuthPacketConstructionTests" name="testConstructorDefaults" time="0"/>
<testcase classname="pyrad.tests.testPacket.AuthPacketConstructionTests" name="testConstructorIgnoredParameters" time="0"/>
<testcase classname="pyrad.tests.testPacket.AuthPacketConstructionTests" name="testConstructorWithAttributes" time="0"/>
<testcase classname="pyrad.tests.testPacket.AuthPacketConstructionTests" name="testNamedConstructor" time="0"/>
<testcase classname="pyrad.tests.testPacket.AuthPacketTests" name="testCreateReply" time="0"/>
<testcase classname="pyrad.tests.testPacket.AuthPacketTests" name="testPwCryptEmptyPassword" time="0"/>
<testcase classname="pyrad.tests.testPacket.AuthPacketTests" name="testPwCryptPassword" time="0"/>
<testcase classname="pyrad.tests.testPacket.AuthPacketTests" name="testPwCryptSetsAuthenticator" time="0"/>
<testcase classname="pyrad.tests.testPacket.AuthPacketTests" name="testPwDecryptEmptyPassword" time="0"/>
<testcase classname="pyrad.tests.testPacket.AuthPacketTests" name="testPwDecryptPassword" time="0"/>
<testcase classname="pyrad.tests.testPacket.AuthPacketTests" name="testRequestPacket" time="0"/>
<testcase classname="pyrad.tests.testPacket.AuthPacketTests" name="testRequestPacketCreatesAuthenticator" time="0"/>
<testcase classname="pyrad.tests.testPacket.AuthPacketTests" name="testRequestPacketCreatesID" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketConstructionTests" name="testBasicConstructor" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketConstructionTests" name="testConstructWithDictionary" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketConstructionTests" name="testConstructorIgnoredParameters" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketConstructionTests" name="testConstructorWithAttributes" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketConstructionTests" name="testNamedConstructor" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testAddAttribute" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testAttributeAccess" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testAttributeValueAccess" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testCreateAuthenticator" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testCreateReply" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testDecodePacketWithAttribute" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testDecodePacketWithBadAttribute" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testDecodePacketWithEmptyAttribute" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testDecodePacketWithEmptyPacket" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testDecodePacketWithInvalidLength" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testDecodePacketWithMultiValuedAttribute" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testDecodePacketWithPartialAttributes" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testDecodePacketWithTooBigPacket" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testDecodePacketWithTwoAttributes" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testDecodePacketWithVendorAttribute" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testDecodePacketWithoutAttributes" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testDelItem" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testEncodeKey" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testEncodeKeyValues" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testGenerateID" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testHasKey" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testHasKeyWithUnknownKey" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testKeys" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testPktDecodeVendorAttribute" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testPktEncodeAttribute" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testPktEncodeAttributes" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testRawAttributeAccess" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testReplyPacket" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testVendorAttributeAccess" time="0"/>
<testcase classname="pyrad.tests.testPacket.PacketTests" name="testVerifyReply" time="0"/>
<testcase classname="pyrad.tests.testPacket.UtilityTests" name="testGenerateID" time="0"/>
<testcase classname="pyrad.tests.testProxy.OtherTests" name="testProcessInput" time="0"/>
<testcase classname="pyrad.tests.testProxy.OtherTests" name="testProcessInputNonProxyPort" time="0"/>
<testcase classname="pyrad.tests.testProxy.ProxyPacketHandlingTests" name="testHHandleProxyPacketHandlesWrongPacket" time="0"/>
<testcase classname="pyrad.tests.testProxy.ProxyPacketHandlingTests" name="testHandleProxyPacketSetsSecret" time="0"/>
<testcase classname="pyrad.tests.testProxy.ProxyPacketHandlingTests" name="testHandleProxyPacketUnknownHost" time="0"/>
<testcase classname="pyrad.tests.testProxy.SocketTests" name="testProxyFd" time="0"/>
<testcase classname="pyrad.tests.testServer.AcctPacketHandlingTests" name="testHandleAcctPacket" time="0"/>
<testcase classname="pyrad.tests.testServer.AcctPacketHandlingTests" name="testHandleAcctPacketUnknownHost" time="0"/>
<testcase classname="pyrad.tests.testServer.AcctPacketHandlingTests" name="testHandleAcctPacketWrongPort" time="0"/>
<testcase classname="pyrad.tests.testServer.AuthPacketHandlingTests" name="testHandleAuthPacket" time="0"/>
<testcase classname="pyrad.tests.testServer.AuthPacketHandlingTests" name="testHandleAuthPacketUnknownHost" time="0"/>
<testcase classname="pyrad.tests.testServer.AuthPacketHandlingTests" name="testHandleAuthPacketWrongPort" time="0"/>
<testcase classname="pyrad.tests.testServer.OtherTests" name="testAcctProcessInput" time="0"/>
<testcase classname="pyrad.tests.testServer.OtherTests" name="testAuthProcessInput" time="0"/>
<testcase classname="pyrad.tests.testServer.OtherTests" name="testCreateReplyPacket" time="0"/>
<testcase classname="pyrad.tests.testServer.RemoteHostTests" name="testNamedConstruction" time="0"/>
<testcase classname="pyrad.tests.testServer.RemoteHostTests" name="testSimpleConstruction" time="0"/>
<testcase classname="pyrad.tests.testServer.ServerConstructiontests" name="testBindDuringConstruction" time="0"/>
<testcase classname="pyrad.tests.testServer.ServerConstructiontests" name="testParameterOrder" time="0"/>
<testcase classname="pyrad.tests.testServer.ServerConstructiontests" name="testSimpleConstruction" time="0"/>
<testcase classname="pyrad.tests.testServer.ServerRunTests" name="testRunIgnoresPacketErrors" time="0"/>
<testcase classname="pyrad.tests.testServer.ServerRunTests" name="testRunIgnoresPollErrors" time="0"/>
<testcase classname="pyrad.tests.testServer.ServerRunTests" name="testRunIgnoresServerPacketErrors" time="0"/>
<testcase classname="pyrad.tests.testServer.ServerRunTests" name="testRunInitializes" time="0"/>
<testcase classname="pyrad.tests.testServer.ServerRunTests" name="testRunRunsProcessInput" time="0"/>
<testcase classname="pyrad.tests.testServer.SocketTests" name="testBind" time="0"/>
<testcase classname="pyrad.tests.testServer.SocketTests" name="testGrabPacket" time="0"/>
<testcase classname="pyrad.tests.testServer.SocketTests" name="testPrepareSocketAcctFds" time="0"/>
<testcase classname="pyrad.tests.testServer.SocketTests" name="testPrepareSocketAuthFds" time="0"/>
<testcase classname="pyrad.tests.testServer.SocketTests" name="testPrepareSocketNoFds" time="0"/>
<testcase classname="pyrad.tests.testTools.EncodingTests" name="testAddressDecoding" time="0"/>
<testcase classname="pyrad.tests.testTools.EncodingTests" name="testAddressEncoding" time="0"/>
<testcase classname="pyrad.tests.testTools.EncodingTests" name="testDateDecoding" time="0"/>
<testcase classname="pyrad.tests.testTools.EncodingTests" name="testDateEncoding" time="0"/>
<testcase classname="pyrad.tests.testTools.EncodingTests" name="testDecodeFunction" time="0"/>
<testcase classname="pyrad.tests.testTools.EncodingTests" name="testEncodeFunction" time="0"/>
<testcase classname="pyrad.tests.testTools.EncodingTests" name="testIntegerDecoding" time="0"/>
<testcase classname="pyrad.tests.testTools.EncodingTests" name="testIntegerEncoding" time="0"/>
<testcase classname="pyrad.tests.testTools.EncodingTests" name="testInvalidAddressEncodingRaisesTypeError" time="0"/>
<testcase classname="pyrad.tests.testTools.EncodingTests" name="testInvalidDataEncodingRaisesTypeError" time="0"/>
<testcase classname="pyrad.tests.testTools.EncodingTests" name="testInvalidIntegerEncodingRaisesTypeError" time="0"/>
<testcase classname="pyrad.tests.testTools.EncodingTests" name="testInvalidStringEncodingRaisesTypeError" time="0"/>
<testcase classname="pyrad.tests.testTools.EncodingTests" name="testStringDecoding" time="0"/>
<testcase classname="pyrad.tests.testTools.EncodingTests" name="testStringEncoding" time="0"/>
<testcase classname="pyrad.tests.testTools.EncodingTests" name="testUnknownTypeDecoding" time="0"/>
<testcase classname="pyrad.tests.testTools.EncodingTests" name="testUnknownTypeEncoding" time="0"/>
<testcase classname="pyrad.tests.testTools.EncodingTests" name="testUnsignedIntegerEncoding" time="0"/>
</testsuite> |
|
This stackoverflow discussion might have useful information. |
|
Thanks @wichert. I'm going to have to have another look at this. |
|
If you need another example: I put some zope testrunner output online as well. One thing that testrunner is does create a separate file for each source file containing tests so you end up with lots of XML files. |
|
Can you provide a status update for this ticket? |
|
I'm really sorry, Wichert. I managed to drop this somewhere along the line. |
|
I'll give it a try. I was still using an older version of Catch, so I'll have to test upgrading first. The first thing I see is lots of new warnings produced by catch and a compile error. so it doesn't appear to be a trivial upgrade. |
|
I think we need to think a bit about how to structure the output. As an example lets assume tests that are setup like this: TEST_CASE("ClassA", "Short description of class A") {
SECTION("methodOne", "Tests for method one") {
SECTION("situation-1", "What happens if XYZ") {
REQUIRE(...);
REQUIRE(...);
REQUIRE(...);
}
SECTION("situation-2", "What happens if XYZ") {
REQUIRE(...);
REQUIRE(...);
REQUIRE(...);
}
}
SECTION("methodTwo", "Tests for method two") {
SECTION("situation-1", "What happens if XYZ") {
REQUIRE(...);
REQUIRE(...);
REQUIRE(...);
}
SECTION("situation-2", "What happens if XYZ") {
REQUIRE(...);
REQUIRE(...);
REQUIRE(...);
}
}
}
TEST_CASE("ClassB", "Short description of class B") {
SECTION("methodOne", "Tests for method one") {
// Repeat similar structure as for classAwhen reporting results for these tests in JUnit format we run into one problem: Catch supports arbitrary nesting of sections, while JUnit only supports two levels (testsuite -> testcase). I suggest that the simplest thing to do here is to only take the two top levels of Catch (TEST_CASE and top level SECTION). That results in this JUnit structure: <testsuites>
<testsuite name="ClassA">
<testcase name="Tests for method one" classname="methodOne" />
<testcase name="Tests for method two" classname="methodTwo" />
</testsuite>
<testsuite name="ClassB">
<testcase name="Tests for method one" classname="methodOne" />
<testcase name="Tests for method two" classname="methodTwo" />
</testsuite>
</testsuites>(ignoring the mandatory attributes such as time, tests, failures, etc.). Looking at the data passed to a Catch reporter this does not match correctly: the top level there is a group, but I never see more than one group being generated. I'm not sure if that is due to me not knowing how to make groups, if that is an not fully implemented feature in Catch or a design flaw (unused group level). Can you provide some guidance how I should proceed with this? |
|
Hi Phil. Can you spare a few minutes to give me a few tips so I can try to fix this? |
|
Hi Phil. Is this still on your radar? |
|
Sorry Wichert, I've still not really caught up. On 13 Mar 2012, at 09:05, Wichert Akkerman wrote:
|
|
Hi Phil. I'm still willing to poke at this, but need some input from you. Is this issue still on your radar? |
|
Hey Wichert, Thanks for your patience. I'm really sorry I've not been getting back to this - and thanks for your help. As for your the question you raised before (sorry I didn't respond before - that was inbox syndrome): The way I have been approaching it is that each isolated test case run (one for each leaf section) is a JUnit "test". The name can be constructed from the test case name + section name(s) (I need to formalise that naming scheme too). JUnit test suites are then each set of tests that are matched by the filters provided by a single -t switch on the command line (if no -t is provided then there is just a single suite). Bear in mind that -t can have multiple filters. So, e.g: -t abc/def* ghi/jkl* -t random/1 random/2 Gives us two suites - one named, "abc/def* ghi/jkl*", and one named, "random/1 random/2". Does that make sense? |
|
This matches the naming scheming, which I admint I have never understood in catch. I am always struggling with two things:
That aside there is a problem with junit XML support: Catch supports nested test cases which does not map to the junit scheme which only supports three levels: package, class and test. Trying force Catch structure into that I would expect something like this:
or alternatively:
The latter matches python (and I'm guessing java) better, but I am not sure if you can get the source filename in a useful way. |
|
The idea is that the test name is a short, hierarchical, name - something like: "stuff/sub stuff/details" - so all "stuff" tests can be grouped, and all "stuff/sub stuff" tests can be grouped. Same with sections, although the hierarchical part is less common as they are already in a hierarchy - so I use descriptions even less there. If you have used hierarchical test case naming (ignore sections for a moment) then you might have tests like: "a/b/c" Now you can run the first three with The idea has always been that this ability would seamlessly extend to sections too - so a section, Nonetheless, the ability to create groups on-the-fly using wildcards (especially with hierarchical naming) has been useful - and has got even more useful just recently as I have added prefix wildcards ( So a command line like:
Would run two groups. The first is Two features that are coming: named groups and tags (non hierarchical matching) should make this approach more powerful. So my scheme was to map groups to Junit suites. |
|
What I am struggling with is the need to prefix fhings manually. To TEST_CASE("MyClass") {
TEST_CASE("SomeMethod") {
SECTION("situation-one") {
....
}
SECTION("situation-two") {
....
}
}
}I would expect to be able to tell Catch to run tests for MyClass using TEST_CASE("MyClass") {
TEST_CASE("MyClas/SomeMethod") {
SECTION("MyClass/SomeMethod/situation-one") {
....
}
SECTION("MyClass/SomeMethod/situation-two") {
....
}
}
} |
|
I'm not sure where the outer TEST_CASE is coming from. It is true the test cases that logically belong together require a common prefix for the hierarchical matching to work. But it's precisely that prefixing that logically groups them together. Sections do not require prefixes, however. So I believe your example would be written something like: Which is not so bad. Now you can run all MyClass tests with:
And everything in that first test case with:
Unfortunately, at time of writing, you cannot selectively run Sections within a test case. There are some issues around this that make it not straightforward, but I believe I should be able to get it working to some approximation (the main issue is that discovery of sections only occurs as the test case is running). Does that clarify anything at all? Is it the section selection that you are particularly missing? |
|
I misremembered by code - I was nesting SECTIONs instead of TEST_CASEs. That makes my code look like this: TEST_CASE("MyClass") {
SECTION("SomeMethod") {
SECTION("situation-one") {
....
}
SECTION("situation-two") {
....
}
}
}I still do not see why I need to repeat a prefix in the section name when I already create an explicit nesting level in the code. Is there no way to avoid that? |
|
Section selection is certainly something that I am missing. Especially when debugging I often want to run a single (leaf) section so I can test for unintended side-effects and side breakpoints easily without having to worry about other tests triggering them. |
|
Hey, I just wanted to chime in here on this conversation of two years :) I experienced the same error in Jenkins that you were experiencing Wichert. The NullPointerException because of the output format of the JUnit reporter. I was able to figure out why Jenkins is crashing, and I have an ugly workaround that lets Jenkins continue to run, but it doesn't solve this issue. Currently, in the JunitReporter class, there is a function that looks like this: Using the above, the output from the JunitReporter looks like the following: After some trial and error, I found that what was making Jenkins crash was the fact that the testcase element did NOT have the classname attribute. This is the line of code in the OutputTestCases() function of the JunitReporter class that causes the trouble: In my case, this line was not actually printing the classname. Perhaps it->m_className is null? When I switched out that line for a random string of "foo", it printed. The modified OutputTestCases() function looks like this: With the above change the JunitReporter output looks like this: Jenkins seems to be okay with this, though it displays the SimpleTest result as if it were a part of the class foo, which is the ugly part I was talking about (but at least it didn't crash!). I hope this helps you debug the JunitReporter. Great work on Catch Phil! It is VERY easy to use, which I really appreciate! :) |
|
I hadn't realised the missing class name attribute was the only thing (or, at least, the key thing) stopping this from working! (I've still not had a chance to get a Jenkins/ Hudson set-up going to try it for myself). I've plumbed that attribute in now (it was always using an empty string before). I realise there are some other attributes that are still set to "tbd". I don't know how important they are to getting this working too. I've committed those changes to the Interation branch. Would appreciate if you (both) could let me know how that works for you. |
|
@wichert I noticed, while replying above, that our previous discussion never fully concluded. But I was bit confused about this: "I still do not see why I need to repeat a prefix in the section name when I already create an explicit nesting level in the code. Is there no way to avoid that?" I don't see any repeated prefix in the last code you posted. |
|
@philsquared are you mixing up this discussion and another issue where we talked about how -s behaves? |
|
No, it was part of this thread: Hard link here: #5 (comment) |
|
I have to admit I don't remember exactly what my thoughts were at the time. Some current thoughts:
|
|
Phil, I just used the catch.hpp file in the Integration branch and Jenkins was quite happy with the output! I tested it with different successful and unsuccessful test cases, and it seems to be working just fine. Thank you for taking time to fix this problem! The other tbd values still need work, but at least Jenkins can read and display the output from the JunitReporter. Thanks again for your help, and again great job on Catch! |
|
That's great! Thanks for letting me know, Derek. @wichert - if it still doesn't work at all for you please reopen. For any othe JUnit related issues now please raise a new issue. |
|
Sorry about that. I've adjusted the permissions so you should be able to access it now. |
|
Hello Phil, |
|
May I add another point to my wish list again? Test failures may lead to several results:
The first case works rather well now. To catch the second case, we introduced a script that checks the exit code. The Main() routine of CATCH was wrapped such that a zero exit code is returned if any test cases failed. This is different if an uncatched exception or even a segmentation fault appears. In this case the exit code is scanned, and the script generates its own XML for JUnit/Jenkins that contains the error information on 'calling the executable'. With the new strategy of catching the std output, however, I can not track the output in case of a segfault since it is caught by CATCH. I do not know if it is possible to 'move' the std output if no segfault happens and to 'copy' the std output if a segfault happens. If this is not possible, maybe you can simply add an option to send output to both, the XML and to console? It would also be helpful if the console output would separate the std output of the individual test cases and sections somehow (eg. "Now start test case 'xy/z'"). Best regards |
|
Hi Phil! Today I encountered a small bug in the JUnit reporter: On some occasions (Eg. if CHECK_THROWS fails) a wrong XML is written, where the element is empty, eg. < message=""> is written instead of . I've seen in the switch-case-block you simply do nothing for certain outcomes. , I changed it to: Best regards |
|
Hmm... Sorry @SebDyn, I don't believe I saw your last two comments before (I suspect because of the way my mail client does threading). For your segfault issue I intend a more comprehensive handling, as mentioned recently in: #160. For the CHECK_THROWS issue I can confirm that DidntThrowException should have handled (the others not, as they are flags that are just there to stop warnings). |
|
I tested this again today with the current (v0.9 build 38) single include from the integration branch. It works, but the created structure is not very optimal:
This completely wastes two levels of test structure which would be very useful to use, and if you use the same test case names in multiple source files (for example I would suggest to change the output as follows:
|
|
@wichert thanks for the continuing feedback. I recently (finally!) got Jenkins installed on my laptop (and I'm told it's now installed on the server where I run my CI builds (thanks Paul) - but I've not had a chance to set that up yet). For a while I was thinking it's so bad that the feedback is much nicer if you just run the console reporter and capture that! But I will go back and improve things. As for your specific suggestions: "use the filename as the package name"
"move TEST_CASE to the send class level"
"expose the top level SECTION inside a test case as the test name."
|
|
While junit is the most popular output format it is of course not the only one. Looking at the list of Jenkins plugins there are a bunch of alternatives: Cpptest, Gallio/MbUnit, JavaTest, JSUnit, NUnit and xUnit which supports a whole bunch of formats. Perhaps one of those will be a better fit. |
|
It looks like all those plugins just convert various format to junit internally, so you are still bound to the three-level package/class/test structure. |
|
Therein, as they say, lies the rub. I wouldn't hold Catch's own XML format up as a shining paradigm of cross framework/ language support either (but maybe we should do something about that). But it has a few characteristics that are in the right direction already - especially that error counts are given in separate elements and at the end - so the report can be streamed as it is being written to. |
|
Hello, here are my 5 cents since I do not fully agree with Wichert's ideas I do not recommend to use the source file name for the class name. In my case I only have a single test case per source file… In my environment I post-process the JUnit XML and replace the class name by the name of the test executable (without path). Each executable collects a set of test cases grouping them semantically in a single executable. So far I could live with "package name=(root)". Here is my recommendation:
I agree with having the SECTION definitions as part of the test names, eg.: could result in failed tests Then an important note on a bug (feature?) in Jenkins: Even if multiple tests failed within a single test case, Jenkins only displays ONE of them! As a workaround all failed tests should be concatenated into a single section. It would be extremely useful to have the times used by a test case available in the JUnit output. It would be nice if CATCH detects if any "boost" time headers were included and use the boost time functions if available in catch.hpp: Best regards PS: I am sorry for not replying, Phil. I use an older revision of CATCH in a production environment and currently I do not want to upgrade. On May 14, 2013, at 8:46 AM, Phil Nash wrote:
|
|
Comment by SebDyn above reformatted: Hello, here are my 5 cents since I do not fully agree with Wichert's ideas. I do not recommend to use the source file name for the class name. In my case I only have a single test case per source file… In my environment I post-process the JUnit XML and replace the class name by the name of the test executable (without path). Each executable collects a set of test cases grouping them semantically in a single executable. So far I could live with "package name=(root)". Here is my recommendation:
I agree with having the SECTION definitions as part of the test names, eg.: could result in failed tests Then an important note on a bug (feature?) in Jenkins: Even if multiple tests failed within a single test case, Jenkins only displays ONE of them! As a workaround all failed tests should be concatenated into a single section. It would be extremely useful to have the times used by a test case available in the JUnit output. It would be nice if CATCH detects if any "boost" time headers were included and use the boost time functions if available in catch.hpp: Best regards PS: I am sorry for not replying, Phil. I use an older revision of CATCH in a production environment and currently I do not want to upgrade. On May 14, 2013, at 8:46 AM, Phil Nash wrote:
|
|
While I see some of the points from @SebDyn I have to disagree with others. I indeed also tend to have very few and often just one test case per source file, so using the test case name only and ignoring the file name would be fine for me. My current codebases are not large enough to warrant multiple CATCH test runners (I do have others, but those are python-based for python wrappers) so using the executable name did not occur to me. For large codebases I can see that that could be useful. I wonder what percentages of code bases is large enough to warrant that. For my situations it would be a shame to loose that hierarchy level. |
|
The code base I'm currently working on by day has about 12 Catch test executables (plus a few NUnit executables). I think process name is the logical fit for the package name. There's possibly scope for making that configurable, though. When I get back to looking at this again I'll take that into consideration. Thanks for your additional comments @SebDyn - I'll ruminate on them more when I'm back in the right context to think about it. |
|
Totally awesome library - so novel and unique and usable! I just hit a little snag with the junit reporter causing UnitTH to crash - there was a missing context.testGroupStarting( "", 1, 1 ); // deprecated?It would be great if we could group tests into some higher level grouping,e.g. TEST_SUITE("Chugger") {
TEST_CASE(...) {
};
TEST_CASE(...) {
}
}(I realize this is outside the scope of this issue, which is the junit reporter, but having the TEST_SUITE defined will then provide a name for the |
|
The test grouping is also requested in #320. |
|
Yes, by me - sorry for the cross post, but the two are related. I'm finding tags a bit of a pain, and I often forget. For example, I just made unit tests for a PRNG, in random.cpp. Each test then has as a minimum one tag |
|
I've moved this part of the discussion back over to #320 |
|
Is there anything left for this one? |
I just tried recent Catch 1.1 build 1 and frankly I need to reply with: No. It is - again - not really working. Here is the reason:
I use Jenkins 1.596.2 (stable) with JUnit reporter. The interesting thing is: Jenkins does not return with an error. It just does not display the test results when using the current CATCH output. |
|
Ok, it seems that Jenkins has a bug somewhere. The "name" keyword ist a mandatory field in JUnit 6 standard. We continue investigating this issue. |
|
The fields with "tbd" really need to go or be fixed. Jenkins' Junit plugin performs duplication checks on inputs by comparing the name, id and timestamp fields (https://github.com/jenkinsci/junit-plugin/blob/master/src/main/java/hudson/tasks/junit/TestResult.java#L240). If the timestamp field would be missing, it would work correctly (strictEq), but the dummy value "tbd" matches the check and Jenkins throws away all other test results. |
|
Timestamp should now be filled with proper UTC-based time, at the time of writing the results, and most other issues raised in this thread be addressed as well. If there are some issues remaining, please open a new issue for them. |
It should be possible to obtain xml reports that follow the schema for JUnit - for consumption by third party tools
The text was updated successfully, but these errors were encountered: