Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JUnit formatter incorrectly reports the total number of tests run #6772

Closed
AlexHowansky opened this issue Mar 9, 2022 · 5 comments · Fixed by phpstan/phpstan-src#1722
Closed

Comments

@AlexHowansky
Copy link

AlexHowansky commented Mar 9, 2022

Bug report

Using v1.4.8

When logging via the JUnit formatter, the tests attribute in the testsuite element gets set to the number of failures. It should get set to the total number of tests performed.

Code snippet that reproduces the problem

vendor/bin/phpstan --error-format=junit

This is the log for a run with 50 tests, 1 of which fails:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite failures="1" name="phpstan" tests="1" xmlns:xsi="...">

Expected output

Via this JUnit doc page:

Child elements: <testsuite>
Attributes:
  id: The ID of the scan.
  name: The label of the scan.
  tests: The total number of rules that were applied.
  failures: The total number of rule violations.
  time The time that was required to process all the rules.

Note the number of tests is known to be 50 because of the progress meter:

50/50 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%

The expected output is thus:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite failures="1" name="phpstan" tests="50" xmlns:xsi="...">

The main side effect of this issue is that when all tests pass, the log contains tests="0" and a CI/CD pipeline scanning that log reports that nothing was done. For example, here is BitBucket showing the "Static Analysis" step which should have "50 tests passed":

image

Did PHPStan help you today? Did it make you happy in any way?

@mergeable
Copy link

mergeable bot commented Mar 9, 2022

This bug report is missing a link to reproduction on phpstan.org.

It will most likely be closed after manual review.

@ondrejmirtes
Copy link
Member

ondrejmirtes commented Mar 9, 2022

What number would you expect there? I can't think of anything that would make sense.

@AlexHowansky
Copy link
Author

AlexHowansky commented Mar 10, 2022

The number that the progress bar displays seemed the obvious choice to me. What does that normally represent, the number of files processed?

@mcaskill
Copy link

mcaskill commented Sep 12, 2022

mglaman/phpstan-junit and Psalm both report the suite as having 1 test if there are no failures/errors (links point to their respective code snippets).

Below is a screen capture demonstrating Psalm's "1 tests passed".

Screen capture of Psalm JUnit test report in Bitbucket Pipelines interface

mcaskill added a commit to mcaskill/phpstan-src that referenced this issue Sep 12, 2022
Resolves phpstan/phpstan#6772

If `AnalysisResult` has no errors, mark `<testsuite>` as having 1 test in order to be considered as having passed.
@github-actions
Copy link

github-actions bot commented Oct 16, 2022

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 16, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants