Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Invalid XML schema for <properties> tags in JUnit reports #5202

Closed
danilomendesdias opened this issue May 3, 2019 · 2 comments

Comments

Projects
None yet
2 participants
@danilomendesdias
Copy link

commented May 3, 2019

The problem:

JUnit breaks when it reads an XML generated by pytest if plugins make use of record-property. This behavior happens with newer versions of hypothesis (HypothesisWorks/hypothesis#1935).

[xUnit] [ERROR] - The result file '/somewhere/tests/pytests.xml' for the metric 'JUnit' is not valid. The result file has been skipped.

In fact, as already mentioned in #1126 (comment), record-property is adding <properties> inside <testcase> which seems to be wrong (it should be inside <testsuite>). See: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd .

It happens with all junit families.

Reproducing:

$ pip list
Package        Version 
-------------- --------
apipkg         1.5     
atomicwrites   1.3.0   
attrs          19.1.0  
certifi        2019.3.9
execnet        1.6.0   
hypothesis     4.18.3  
more-itertools 4.3.0   
pip            19.1    
pluggy         0.9.0   
py             1.8.0   
pytest         4.4.1   
pytest-forked  1.0.2   
pytest-xdist   1.28.0  
setuptools     41.0.1  
six            1.12.0  
wheel          0.33.1 

test_xml_generation.py

from hypothesis import given, strategies


@given(x=strategies.integers(1, 10,))
def test_xml_generation(x):
    assert 1 <= x <= 10
$ pytest --junitxml=report.xml

report.xml

<?xml version="1.0" encoding="utf-8"?>
<testsuite errors="0" failures="0" name="pytest" skipped="0" tests="1" time="0.211">
    <testcase classname="test_xml_generation" file="test_xml_generation.py" line="3" name="test_xml_generation"
              time="0.074">
        <properties>
            <property name="hypothesis-stats"
                      value="[&apos;test_xml_generation.py::test_xml_generation:&apos;, &apos;&apos;, &apos;  - 100 passing examples, 0 failing examples, 0 invalid examples&apos;, &apos;  - Typical runtimes: &lt; 1ms&apos;, &apos;  - Fraction of time spent in data generation: ~ 49%&apos;, &apos;  - Stopped because settings.max_examples=100&apos;, &apos;&apos;]"/>
        </properties>
    </testcase>
</testsuite>

I was trying to create a PR to fix this, but when I saw https://github.com/pytest-dev/pytest/blob/7dcd9bf5add337686ec6f2ee81b24e8424319dba/src/_pytest/junitxml.py code I realized that what is needed to do could have more implications that I though. I think that nobody uses this feature with JUnit (as it breaks) and removing that is something to think about.

@danilomendesdias

This comment has been minimized.

Copy link
Author

commented May 3, 2019

@nicoddemus

This comment has been minimized.

Copy link
Member

commented May 3, 2019

Thanks @danilomendesdias!

nicoddemus added a commit to nicoddemus/pytest that referenced this issue May 3, 2019

Emit a warning for record_property when used with xunit2
"property" elements cannot be children of "testsuite" according to the schema, so it is incompatible with xunit2

Related to pytest-dev#5202

nicoddemus added a commit to nicoddemus/pytest that referenced this issue May 3, 2019

Emit a warning for record_property when used with xunit2
"property" elements cannot be children of "testsuite" according to the schema, so it is incompatible with xunit2

Related to pytest-dev#5202

nicoddemus added a commit to nicoddemus/pytest that referenced this issue May 3, 2019

Introduce record_testsuite_property fixture
This exposes the functionality introduced in fa6acdc as a session-scoped fixture.

Plugins that want to remain compatible with the `xunit2`
standard should use this fixture instead of `record_property`.

Fix pytest-dev#5202

nicoddemus added a commit to nicoddemus/pytest that referenced this issue May 5, 2019

Emit a warning for record_property when used with xunit2
"property" elements cannot be children of "testsuite" according to the schema, so it is incompatible with xunit2

Related to pytest-dev#5202

nicoddemus added a commit to nicoddemus/pytest that referenced this issue May 10, 2019

Introduce record_testsuite_property fixture
This exposes the functionality introduced in fa6acdc as a session-scoped fixture.

Plugins that want to remain compatible with the `xunit2`
standard should use this fixture instead of `record_property`.

Fix pytest-dev#5202

@nicoddemus nicoddemus closed this May 11, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.