Unexpected scenario

[dfrancoed]

Hi guys, I'm following the implementation of the custom data source with the interface ITestDataSource as is described in the example but I found that the test method is returning more scenarios than the expected.

See the attachment.

So, Why I'm getting a 3rd one scenario without parameters if the GetData method only specifies two?

[ShreyasRmsft]

@dfrancoed when getting results for data driven tests or custom data source tests the parent test case result is printed first followed by the child tests results.

ex:
[1, 2]
[3, 4]
testmethod(int a, int b) {}

Results:
testmethod() passed
testmethod(1, 2) passed
testmethod(3, 4) passed

if all of them passed

Results:
testmethod() failed
testmethod(1, 2) passed
testmethod(3, 4) failed

if one failed etc. etc.

This is done so that there is an aggregate parent result for data driven tests which is needed when displaying the results in a hierarchical fashion (say a tree view) in pretty reporters.

This is expected behavior and is as designed.

[dfrancoed]

So in my understanding, this behavior is concerning to the adapter of vstest who's reporting 3 scenarios rather than 2.

That's wrong because affects the stats of the test executions.

a) I mean the hierarchical way is cool but what about affecting the stats?

b) Or what about if I want to know only the number of test cases executed rather than the number of scenarios?

What do you think could be a solution to avoid to sum the tree header in the number of total scenarios. Regards!

[ShreyasRmsft]

@PBoraMSFT for answering the above question

[neomarck]

@dfrancoed when getting results for data driven tests or custom data source tests the parent test case result is printed first followed by the child tests results.

ex:
[1, 2]
[3, 4]
testmethod(int a, int b) {}

Results:
testmethod() passed
testmethod(1, 2) passed
testmethod(3, 4) passed

if all of them passed

Results:
testmethod() failed
testmethod(1, 2) passed
testmethod(3, 4) failed

if one failed etc. etc.

This is done so that there is an aggregate parent result for data driven tests which is needed when displaying the results in a hierarchical fashion (say a tree view) in pretty reporters.

This is expected behavior and is as designed.

Hi, with your example in mind, in VSTS, when you execute a build pipeline that runs test cases, the tests statistics (in VSTS) are not displaying aggregate results, so you will have 3 test cases executed, all at the same level, but the truth is that you only run 1 test case but with two iterations.

This is what I'm facing now in VSTS and now my test results are not reliable anymore. Even with the results that you can see in console, at the end it says it executed 3 tests, but that's not true.

I hope there is a workaround for this issue, maybe there is a way of making a setup in VSTS build pipeline to properly recognize the aggregated results.
Thanks.

[wujysh]

+1
I don't think the output format like above is expected. People reading the log will get confused.

• Can not find the relationship between the parent test and its variations.
• The statistics (total, passed and failed) are not right, which don't reflect the actual situation.

@ShreyasRmsft Is there a way to recover to the behavior of MSTest v1. Or can you tell me the code that controls this display and counting logic? Thanks.

[ShreyasRmsft]

@wujysh for the console logger i think perhaps the counting logic like you said can be changed. This is not a high priority item in our backlog right now but we will accept contributions. Please feel free to contribute.

[ShreyasRmsft]

@neomarck for issues seeing hierarchical results on vsts please refer to #488 (comment)

@wujysh for the console logger i think perhaps the counting logic like you said can be changed. This is not a high priority item in our backlog right now but we will accept contributions. Please feel free to contribute.