Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

change 'Tests Run' and 'Passed' to reflect the number of testcases instead of the number of assertions #180

Closed
peetkes opened this issue Aug 10, 2023 · 2 comments
Milestone

Comments

@peetkes
Copy link
Contributor

peetkes commented Aug 10, 2023

In the GUI the Suite overview gives

  1. Total Test Count, reflects the number of testcases
  2. Tests Run, reflects the number of assertions
  3. Passed, reflects the number of successful assertions
  4. Failed, the number of failed testcases

A suite can have multiple testcases and each testcase can have multiple assertions.
But as soon as a testcase has a single failed assertion, the whole testcase is flagged as failed regardless of the number of succeeded assertions up until the failure.
So the count of assertions is not that informative.

I would suggest changing the 'Tests Run' to the number of testcases that were executed and 'Passed' to the number of successful run testcases.

This can be changed in test-controller.xqy lines 183, 184 and 185

      attribute total { fn:count($results/test:test) },
      attribute failed { fn:count($results/test:test[test:result[@type='fail']]) },
      attribute passed { fn:count($results/test:test[test:result[@type='success']]) },
@rjrudin
Copy link
Contributor

rjrudin commented Aug 10, 2023

"number of testcases" = the number of files, right? And "Total Test Count" is already capturing that.

What marklogic-unit-test calls a test:test is really an assertion. I think the data makes sense (ignoring other issues about e.g. setup failures counting as failed assertions), we need better labels. I think these would be better labels:

  • "Total Test Count" -> "Test Count" or "Total Tests" = the number of test modules (this is currently including setup files, which is a bug)
  • "Tests Run" -> this could arguably go away, as it's just summing the number of successful and failed assertions. If we keep it, it's "Assertion Count" or "Total Assertions".
  • "Passed" -> "Success Count" or "Succeeded" (could do "Passed", though it bothers me that we're then using "success" and "passed" instead of just one term; I guess to minimize changes, we'd keep this as "Passed").
  • "Failed" -> "Failure Count" or "Failed".

@rjrudin rjrudin added this to the 1.4.0 milestone Aug 10, 2023
peetkes added a commit to peetkes/marklogic-unit-test that referenced this issue Aug 11, 2023
peetkes added a commit to peetkes/marklogic-unit-test that referenced this issue Aug 12, 2023
rjrudin added a commit that referenced this issue Sep 5, 2023
Also fixing a bug where setup/teardown modules were being counted in the number of tests.

Also removed duplication of logic from default.xqy, which is now using the "is" functions in test-controller.xqy. Added a few of these functions for syntactic sugar as well, just to make the code easier to read in default.xqy. 

We unfortunately don't have any tests for the default.xqy module, so this is being tested manually via ml-gradle's unit-test-example project.
@rjrudin
Copy link
Contributor

rjrudin commented Sep 5, 2023

Addressed via #195 .

@rjrudin rjrudin closed this as completed Sep 5, 2023
rjrudin added a commit that referenced this issue Sep 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants