how to stop Fitnesse from reporting SuiteSetup and SuiteTearDown within XML test results

[andykkng]

Hi,

We have been using an old version (ie.2014) of FitNesse for running our automation tests. Since it has been stable, we were not in a hurry to upgrade FitNesse.

The way we ran Fitnesse is through the command line by using the -c command line option as follows:

  java -jar fitnesse-standalone.jar -c "TestPage?test&format=xml"

However, when we switched to the latest Fitnesse version (ie. 20160515), FitNesse XML result files now reported a number of tags, one for each set up page invoked by the tests, e.g. SuiteSetup and SuiteTearDown, inside the xml result file. Also, the tag reports on all "executed" pages as follows:

3 0 0 0

Previously, the finalCount number only reported on the actual test page. Is there a way to turn off this "Suite" like behavior for reporting "Test" results? Or is there a workaround to get FitNesse to report xml result files as it used to?

Any help is greatly appreciated.

Andy

[suratdas]

I also see that. SuiteSetUp and SuiteTearDown are counted as pages (but not SetUp and TearDown) and hence the test count does not match. If we publish in Jenkins, the test result shows extra counts. @amolenaar @jediwhale : Any reason we count SuiteSetUp and SuiteTearDown in test count but not SetUp and TearDown.

[jediwhale]

SuiteSetUp and SuiteTearDown are executed as separate test pages but SetUp and TearDown are sections that are added to existing test pages. I believe that's why they are counted differently.

…

On 2019-07-26 00:34, Surat Das wrote: I also see that. SuiteSetUp and SuiteTearDown are counted as pages (but not SetUp and TearDown) and hence the test count does not match. If we publish in Jenkins, the test result shows extra counts. @amolenaar <https://github.com/amolenaar> @jediwhale <https://github.com/jediwhale> : Any reason we count SuiteSetUp and SuiteTearDown in test count but not SetUp and TearDown. image <https://user-images.githubusercontent.com/9042580/61930424-4a215180-af9b-11e9-951e-1a1e82705e17.png> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#933?email_source=notifications&email_token=AAAJOY6DD4IXSNWNRYAGAZLQBKLGRA5CNFSM4CFT4XTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD23UTLI#issuecomment-515328429>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAJOYZZNQPCSC5YNQ7NISTQBKLGRANCNFSM4CFT4XTA>.

[briceparmentier]

Hello here !

I found this topic which is exactly what I'm also facing...

Is this a "bug" somewhere, or is this a misconfiguration? I agree it is weird to have SuiteSetUp and SuiteTearDown in the tests results... I can understand though that some people may want to have the results of them beside the tests themselves.
This is why I think this could be a lack of configuration somewhere.

Any news on this?

[fhoeben]

I believe no one ever found it problematic enough to dive into this.

(I personally always run a suite on CI and the exact number of tests reported doesn't interest me, test are added/removed all the time, I just want to know all tests passed)

[briceparmentier]

On our side we are reporting to another tool that connects to requirements and provides coverage views and so on, so yeah the number of tests is really important.
Also, I have tried moving the SuiteSetUp and SuiteTearDown one level up the suite I'm running, and then we don't get them in the tests results, but there will be a problem for us as we need several different SuiteSetUp and SuiteTearDown depending on the subsuites...

[tcnh]

Hi Brice, I also ran into this when logging to an external tool. We use a custom TestSystemListener implementation to do so. We overcame this issue by reporting the SuiteSetUp and SuiteTearDown as before/after steps, instead of test cases. Does your tool allow this sort of workaround?

[briceparmentier]

@tcnh I don't think so.
We are using Jenkins as a sequencer to trigger our FitNesse suites, and then we get the results in a JUnit format file.
This file is then eaten by a Jenkins plugin to link with our reporting/requirements management tool and each test is reported to this tool in the given container. The tool is qTest from Tricentis (https://www.tricentis.com/fr/produits/gestion-unifiee-des-tests-qtest/).
So unless we have something that removes the SuiteSetUp and SuiteTearDown from the JUnit file, I'm not sure I will be able to setup something similar to what you are doing...
Thanks for your feedback anyway!