Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JUnitXmlReportPlugin generates invalid testcase name #2949

Closed
Pyppe opened this issue Feb 2, 2017 · 2 comments
Closed

JUnitXmlReportPlugin generates invalid testcase name #2949

Pyppe opened this issue Feb 2, 2017 · 2 comments
Labels
area/test_framework uncategorized Used for Waffle integration

Comments

@Pyppe
Copy link

Pyppe commented Feb 2, 2017

It seems that in certain cases an invalid test-case name is generated in the JUnit XML output.

steps

This is one arbitrary example demonstrating the issue:

  "This example" should {
    "given <VER.123 FOOBAR> yield <ver.123 foobar>" in {
      ok
    }
  }

problem

With this example (using specs2 3.8.6) the generated JUnit XML contains the following test-case:

<testcase classname="org.example.ExampleSpec" name="123 foobar&gt;" time="0.006">
</testcase>

expectation

Now, obviously the correct XML should be as follows:

<testcase classname="org.example.ExampleSpec" name="This example should::given &lt;VER.123 FOOBAR&gt; yield &lt;ver.123 foobar&gt;" time="0.006">
</testcase>

notes

@Pyppe
Copy link
Author

Pyppe commented Feb 3, 2017

Just by looking at the code;

case selector: TestSelector=> selector.testName.split('.').last
looks fishy:

case selector: TestSelector=> selector.testName.split('.').last

@maichler
Copy link

This issue bothers me as well!

If I understand the API documentation of TestSelector correctly, testName() is supposed to just return the test name itself.

IMHO post processing of testName() in JUnitXmlTestsListener is a bug and should be removed.

Any thoughts ?

@eed3si9n eed3si9n added the uncategorized Used for Waffle integration label Sep 18, 2018
dongjoon-hyun pushed a commit to apache/spark that referenced this issue Aug 30, 2019
…enkins's test results

### What changes were proposed in this pull request?

See https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109834/testReport/junit/org.apache.spark.sql/SQLQueryTestSuite/

![Screen Shot 2019-08-28 at 4 08 58 PM](https://user-images.githubusercontent.com/6477701/63833484-2a23ea00-c9ae-11e9-91a1-0859cb183fea.png)

```xml
<?xml version="1.0" encoding="UTF-8"?>
<testsuite hostname="C02Y52ZLJGH5" name="org.apache.spark.sql.SQLQueryTestSuite" tests="3" errors="0" failures="0" skipped="0" time="14.475">
    ...
    <testcase classname="org.apache.spark.sql.SQLQueryTestSuite" name="sql - Scala UDF" time="6.703">
    </testcase>
    <testcase classname="org.apache.spark.sql.SQLQueryTestSuite" name="sql - Regular Python UDF" time="4.442">
    </testcase>
    <testcase classname="org.apache.spark.sql.SQLQueryTestSuite" name="sql - Scalar Pandas UDF" time="3.33">
    </testcase>
    <system-out/>
    <system-err/>
</testsuite>
```

Root cause seems a bug in SBT - it truncates the test name based on the last dot.

sbt/sbt#2949
https://github.com/sbt/sbt/blob/v0.13.18/testing/src/main/scala/sbt/JUnitXmlTestsListener.scala#L71-L79

I tried to find a better way but couldn't find. Therefore, this PR proposes a workaround by appending the test file name into the assert log:

```diff
  [info] - inner-join.sql *** FAILED *** (4 seconds, 306 milliseconds)
+ [info]   inner-join.sql
  [info]   Expected "1	a
  [info]   1	a
  [info]   1	b
  [info]   1[]", but got "1	a
  [info]   1	a
  [info]   1	b
  [info]   1[	b]" Result did not match for query #6
  [info]   SELECT tb.* FROM ta INNER JOIN tb ON ta.a = tb.a AND ta.tag = tb.tag (SQLQueryTestSuite.scala:377)
  [info]   org.scalatest.exceptions.TestFailedException:
  [info]   at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:528)
```

It will at least prevent us to search full logs to identify which test file is failed by clicking filed test.

Note that this PR does not fully fix the issue but only fix the logs on its failed tests.

### Why are the changes needed?
To debug Jenkins logs easier. Otherwise, we should open full logs and search which test was failed.

### Does this PR introduce any user-facing change?
It will print out the file name of failed tests in Jenkins' test reports.

### How was this patch tested?
Manually tested but Jenkins tests are required in this PR.

Now it at least shows which file it is:

![Screen Shot 2019-08-30 at 10 16 32 PM](https://user-images.githubusercontent.com/6477701/64023705-de22a200-cb73-11e9-8806-2e98ad35adef.png)

Closes #25630 from HyukjinKwon/SPARK-28894-1.

Authored-by: HyukjinKwon <gurwls223@apache.org>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
eed3si9n pushed a commit to eed3si9n/sbt that referenced this issue Mar 13, 2020
… it contains dot (sbt#5139)

Fix for JUnitXmlTestsListener removing part of test when it contains dot
Fixes sbt#5114
Fixes sbt#2949
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 6, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area/test_framework uncategorized Used for Waffle integration
Projects
None yet
Development

No branches or pull requests

4 participants