Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better report of failures, append stderr/out. #3450

Merged
merged 1 commit into from Jan 13, 2022

Conversation

sdedic
Copy link
Member

@sdedic sdedic commented Jan 12, 2022

This is a follow-up to #3403 -- I've noticed that the xmllint --xpath '//testsuite/testcase[./failure]/@name' $file | cut -f2 -d '=' only selects the 1st failed test, in case there are more failed cases in the testsuite.

During last hunt for a CI-failing test (that did not fail on local at all), I've also missed stderr/out for the failed suite. And also the actual assert stacktrace, since not all asserts have String message, the stacktrace can help to identify the exact failure.

Now the output is as follows:

=================== JUnit Report Summary / failed tests ===================

org.netbeans.modules.java.lsp.server.protocol.ServerTest
      failed: testCancelProgressHandle
      failed: testAnnotationCompletion

org.netbeans.modules.java.lsp.server.UtilsTest
      failed: testEncode2JSON
      errored: testEscapeScompletionSnippetSpceialChars

====================== JUnit failure details ===============================

Suite: org.netbeans.modules.java.lsp.server.protocol.ServerTest
      testCancelProgressHandle FAILED : 
          junit.framework.AssertionFailedError
                at org.netbeans.modules.java.lsp.server.protocol.ServerTest.testCancelProgressHandle(ServerTest.java:4932)
                at org.netbeans.junit.NbTestCase.access$200(NbTestCase.java:77)
                at org.netbeans.junit.NbTestCase$2.doSomething(NbTestCase.java:476)
                at org.netbeans.junit.NbTestCase$1Guard.run(NbTestCase.java:402)
                at java.lang.Thread.run(Thread.java:748)
      testAnnotationCompletion FAILED :  Test failure.
          junit.framework.AssertionFailedError: Test failure.
                at org.netbeans.modules.java.lsp.server.protocol.ServerTest.testAnnotationCompletion(ServerTest.java:4591)
                at org.netbeans.junit.NbTestCase.access$200(NbTestCase.java:77)
                at org.netbeans.junit.NbTestCase$2.doSomething(NbTestCase.java:476)
                at org.netbeans.junit.NbTestCase$1Guard.run(NbTestCase.java:402)
                at java.lang.Thread.run(Thread.java:748)
Stderr ----------%<----------%<-------------%<-------------%<---------------
....
------------- End suite org.netbeans.modules.java.lsp.server.protocol.ServerTest ------------

I broke several files so that this PR's checks output will show the effect.

Copy link
Member

@mbien mbien left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome! so much better.

@mbien
Copy link
Member

mbien commented Jan 12, 2022

+ thanks for keeping the summary for a quick overview

@sdedic
Copy link
Member Author

sdedic commented Jan 12, 2022

Rebased on master, the testing broken tests dropped.

@sdedic
Copy link
Member Author

sdedic commented Jan 13, 2022

Merging ...there are still a few tests that cause CI to fail, this could help to track the failures down on CI.

@sdedic sdedic merged commit 84bd3ea into apache:master Jan 13, 2022
@sdedic sdedic self-assigned this Jan 13, 2022
@sdedic sdedic added CI Continuous Integration enhancement labels Jan 13, 2022
@neilcsmith-net neilcsmith-net added this to the NB13 milestone Jan 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI Continuous Integration enhancement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants