Polarion support #243

clacroix12 · 2019-06-17T22:45:56Z

Added the ability to mark tests with their respective polarion_ids in order to support posting our test results to Polarion. When we export our test results to a junit xml file (--junit-xml report.xml) any test cases that are marked with polarion_ids will have a custom property set that reflects the value.

Ex.

<properties><property name="polarion-testcase-id" value="OCS-foo"/></properties>

Note that these changes do not add the ability to post the results to Polarion, only to support the tool we will use to take our xml test results and post the results.

mbukatov

This looks good.

Note that polarion-project-id property is mandatory for testsuites element (as noted in docs of polarion ci importer). But the junit xml report starts with testsuite element:

(venv) [ocsqe@localhost ocs-ci]$ xmllint --format report.xml
<?xml version="1.0" encoding="utf-8"?>
<testsuite errors="0" failures="1" name="pytest" skipped="0" tests="2" time="0.125">
  <testcase classname="tests.test_foo" file="tests/test_foo.py" line="2" name="test_foo_bar" time="0.002">
    <properties>
      <property name="polarion-testcase-id" value="FOO-123"/>
    </properties>
  </testcase>
  <testcase classname="tests.test_foo" file="tests/test_foo.py" line="6" name="test_foo_baz" time="0.001">
    <properties>
      <property name="polarion-testcase-id" value="FOO-120"/>
    </properties>
    <failure message="assert 1 == 0   -1   +0">@pytest.mark.polarion_id("FOO-120")
    def test_foo_baz():
&gt;       assert 1 == 0
E       assert 1 == 0
E         -1
E         +0

tests/test_foo.py:9: AssertionError</failure>
  </testcase>
</testsuite>

So the question is if importer will recognize polarion-project-id property in testsuite element (which we could provide in pytest), or whether we need to wrap the report into testsuites element and add the project id there (which could be done in the CI job pushing the report into the importer).

That said, I'm ok with letting this in, as it's not clear at which point we should add polarion-project-id property.

ocsci/pytest_customization/ocscilib.py

petr-balogh · 2019-06-18T15:00:28Z

ocsci/pytest_customization/ocscilib.py

+    Add Polarion ID property to test cases that are marked with one.
+    """
+    for item in items:
+        for marker in item.iter_markers(name="polarion_id"):


I would rather use https://docs.pytest.org/en/latest/reference.html#_pytest.nodes.Node.get_closest_marker WDYT?

Reiterating what we discussed earlier: I think get_closest_marker is a good idea. This could save us in the event that both a test method and the test class it resides in are marked with polarion_ids that conflict. With get_closest_marker we would only use the value from the test level marker and not the class. This is still a situation we don't want to have present in our tests, but we also don't want to post results to the wrong tests in the event it happens. +1 for get_closest_marker.

What if one automated test covers more test cases in Polarion? (not sure, if it is "allowed" or not, but I think, it is possible to have such case...)

Previously it was possible to use something like:

@pytest.mark.polarion_id("FOO1") @pytest.mark.polarion_id("FOO2") @pytest.mark.polarion_id("FOO3") def test_foo(self): ...

But now, it will use only the last ID (FOO3).

On the other side it doesn't make sense to add Polarion ID on class level, because it doesn't make sense to have more automated tests reporting to one test case in Polarion...

@dahorak I'm not sure if the importer will be able to handle this anyway.

petr-balogh · 2019-06-18T15:02:12Z

ocsci/pytest_customization/ocscilib.py

+    for item in items:
+        for marker in item.iter_markers(name="polarion_id"):
+            polarion_id = marker.args[0]
+            item.user_properties.append(("polarion-testcase-id", polarion_id))


We will need to add one more global required parameter under testsuites to junit file. See the documentation:
https://mojo.redhat.com/docs/DOC-1073077#jive_content_id_polarionprojectid_required

Yes, but I would rather not add it for every test case individually.

Yeah like @mbukatov said in his top level comment we might need to wrap the output from our tests in a <testsuites> element as it doesn't exist in the report by default. The property you linked to needs to be a property of <testsuites> not the <testcase> so I wouldn't add it in this loop.

Ok, I've just dig in to how it's done in rhev and you can see one of generated xunit file which works for polarion importer in RHV automation here: https://rhv-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/4.3/job/rhv-4.3-ge-runner-tier1/81/artifact/xunit_output.xml

I see that it can be in testusite under properties as you can see here

<testsuite errors="0" failures="3" name="pytest" skips="42" tests="706" time="68101.387"> <properties> <property name="polarion-custom-isautomated" value="True"/> <property name="polarion-testrun-status-id" value="inprogress"/><property name="polarion-custom-plannedin" value="RHV_4_3"/> <property name="polarion-user-id" value="ci-user"/> <property name="polarion-project-id" value="RHEVM3"/><property name="polarion-response-myproduct" value="rhvm"/> <property name="polarion-testrun-id" value="RHV_4_3_tier1_x86_64"/><property name="polarion-custom-arch" value="x8664"/> </properties> <testcase classname="rhevmtests.compute.sla.cpu_qos.cpu_qos_test.TestQoSAndCpuProfileCRUD" file="rhevmtests/compute/sla/cpu_qos/cpu_qos_test.py" line="60" name="test_a_add_qos" time="11.999502182"><properties><property name="polarion-testcase-id" value="RHEVM-14931"/></properties> <system-out> ...output omitted...

I added a session fixture to handle adding these properties. For now it only adds the polarion-project-id but we can update it to add any others we might need.

vasukulkarni · 2019-06-18T15:26:32Z

This looks good.

Note that polarion-project-id property is mandatory for testsuites element (as noted in docs of polarion ci importer). But the junit xml report starts with testsuite element:
(venv) [ocsqe@localhost ocs-ci]$ xmllint --format report.xml
<?xml version="1.0" encoding="utf-8"?>
<testsuite errors="0" failures="1" name="pytest" skipped="0" tests="2" time="0.125">
  <testcase classname="tests.test_foo" file="tests/test_foo.py" line="2" name="test_foo_bar" time="0.002">
    <properties>
      <property name="polarion-testcase-id" value="FOO-123"/>
    </properties>
  </testcase>
  <testcase classname="tests.test_foo" file="tests/test_foo.py" line="6" name="test_foo_baz" time="0.001">
    <properties>
      <property name="polarion-testcase-id" value="FOO-120"/>
    </properties>
    <failure message="assert 1 == 0   -1   +0">@pytest.mark.polarion_id("FOO-120")
    def test_foo_baz():
&gt;       assert 1 == 0
E       assert 1 == 0
E         -1
E         +0

tests/test_foo.py:9: AssertionError</failure>
  </testcase>
</testsuite>
So the question is if importer will recognize polarion-project-id property in testsuite element (which we could provide in pytest), or whether we need to wrap the report into testsuites element and add the project id there (which could be done in the CI job pushing the report into the importer).

That said, I'm ok with letting this in, as it's not clear at which point we should add polarion-project-id property.

project-id and test-run-id can be set at jenkins level? since they are just one time per testsuite?

clacroix12 · 2019-06-18T17:28:22Z

project-id and test-run-id can be set at jenkins level? since they are just one time per testsuite?

@vasukulkarni That's one possibility as we will already need to wrap the junit.xml in a <testsuites> element anyways. We might as well handle the <testsuites> level properties in the same operation. The other possibility is we format it and add the properties ourselves as a session level finalizer if we have passed the CLI args to export the junit.xml results. Either of which I think can be addressed outside of this PR once we have made some progress with running the actual importer.

petr-balogh · 2019-06-18T20:17:45Z

project-id and test-run-id can be set at jenkins level? since they are just one time per testsuite?

@vasukulkarni That's one possibility as we will already need to wrap the junit.xml in a <testsuites> element anyways. We might as well handle the <testsuites> level properties in the same operation. The other possibility is we format it and add the properties ourselves as a session level finalizer if we have passed the CLI args to export the junit.xml results. Either of which I think can be addressed outside of this PR once we have made some progress with running the actual importer.

I prefer to prepare whole xunit file in ocs-ci as we can then just use it even in some manual upload to polarion without any further changes, and in the future we will need maybe add more properties.

clacroix12 · 2019-06-18T22:55:12Z

I wanted to bring attention to a small issue I'm seeing with these updates. In adding the log statement in ocscilib.py it seems like some of the pytest output gets captured and placed in the <system-out> of one of my test functions in the junit report xml. I'm not exactly sure why and it appears to only happen for the first test case that is executed. I have confirmed that simply commenting out the log statement is enough to get this output to not show up in the junit xml.

    <testcase classname="tests.test_pytest_html.TestPolarionExample"
              file="tests/test_pytest_html.py" line="30"
              name="test_without_marker" time="0.003">
        <system-out>#x1B[1m
            collected 3 items #x1B[0m

            tests/test_pytest_html.py::TestPolarionExample::test_without_marker
        </system-out>
    </testcase>

This isn't a blocker and as such didn't want to hold up this PR for it, but I wanted to call it out and see if anyone would have any ideas as to why this is happening.

mbukatov

Looks good. The only problem is in the default value of project_id, but this could be reconfigured.

conf/ocsci/default_config.yaml

ocs/defaults.py

mbukatov · 2019-06-19T08:39:58Z

This isn't a blocker and as such didn't want to hold up this PR for it, but I wanted to call it out and see if anyone would have any ideas as to why this is happening.

This looks like a peculiarity of the pytest xml report. Logs and output from collecting phase doesn't belong to any test and so the xml report seems to just add it into the 1st one instead of dropping it from the xml report.

I don't think this is a big deal, as in normal operation in CI or any other valid test run, we are not expected to have any warnings reported in pytest hooks. The warning here is for author of the test code to notice something needs to be addressed before merging.

Signed-off-by: Coady LaCroix <clacroix@redhat.com>

…without providing a value Signed-off-by: Coady LaCroix <clacroix@redhat.com>

Signed-off-by: Coady LaCroix <clacroix@redhat.com>

RazTamir · 2019-06-20T15:53:43Z

With this PR is now merged, could you please provide information, what is needed for test cases to include in order to be in sync with Polarion?

clacroix12 requested a review from a team as a code owner June 17, 2019 22:45

clacroix12 mentioned this pull request Jun 17, 2019

Reporting integration #73

Closed

2 tasks

clacroix12 added enhancement New feature or request High Priority High priority issues labels Jun 17, 2019

mbukatov previously approved these changes Jun 18, 2019

View reviewed changes

dahorak previously approved these changes Jun 18, 2019

View reviewed changes

vasukulkarni reviewed Jun 18, 2019

View reviewed changes

ocsci/pytest_customization/ocscilib.py Outdated Show resolved Hide resolved

vasukulkarni previously approved these changes Jun 18, 2019

View reviewed changes

petr-balogh reviewed Jun 18, 2019

View reviewed changes

clacroix12 dismissed stale reviews from vasukulkarni, dahorak, and mbukatov via c663ae5 June 18, 2019 22:44

clacroix12 force-pushed the polarion_support branch from 36d45fc to c663ae5 Compare June 18, 2019 22:44

vasukulkarni previously approved these changes Jun 18, 2019

View reviewed changes

mbukatov previously approved these changes Jun 19, 2019

View reviewed changes

conf/ocsci/default_config.yaml Outdated Show resolved Hide resolved

ocs/defaults.py Outdated Show resolved Hide resolved

clacroix12 dismissed stale reviews from mbukatov and vasukulkarni via ac00be1 June 19, 2019 15:09

mbukatov previously approved these changes Jun 19, 2019

View reviewed changes

clacroix12 added 5 commits June 19, 2019 08:20

Added polarion_id marker to support posting test results to Polarion

6440e95

Signed-off-by: Coady LaCroix <clacroix@redhat.com>

Decorated existing tests with their respective polarion_ids

6569cf4

Signed-off-by: Coady LaCroix <clacroix@redhat.com>

Added autouse session fixture to configure polarion testsuite properties

0da2343

Signed-off-by: Coady LaCroix <clacroix@redhat.com>

Use get_closest_marker and log warning if polarion_id marker is used …

6963747

…without providing a value Signed-off-by: Coady LaCroix <clacroix@redhat.com>

Fixed default polarion project_id

64cfa0d

Signed-off-by: Coady LaCroix <clacroix@redhat.com>

petr-balogh previously approved these changes Jun 19, 2019

View reviewed changes

mbukatov mentioned this pull request Jun 19, 2019

Include rbd into invalid storageclass fixture #250

Merged

Import config module instead of REPORTING section

f2f7120

Signed-off-by: Coady LaCroix <clacroix@redhat.com>

clacroix12 dismissed stale reviews from petr-balogh and mbukatov via f2f7120 June 19, 2019 16:57

clacroix12 force-pushed the polarion_support branch from ac00be1 to f2f7120 Compare June 19, 2019 16:57

vasukulkarni approved these changes Jun 19, 2019

View reviewed changes

petr-balogh approved these changes Jun 19, 2019

View reviewed changes

clacroix12 merged commit bd1e912 into red-hat-storage:master Jun 19, 2019

clacroix12 deleted the polarion_support branch June 19, 2019 19:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Polarion support #243

Polarion support #243

clacroix12 commented Jun 17, 2019

mbukatov left a comment •

edited

Loading

petr-balogh Jun 18, 2019

clacroix12 Jun 18, 2019

clacroix12 Jun 18, 2019

dahorak Jun 19, 2019 •

edited

Loading

mbukatov Jun 19, 2019

petr-balogh Jun 18, 2019

mbukatov Jun 18, 2019

clacroix12 Jun 18, 2019

petr-balogh Jun 18, 2019

clacroix12 Jun 18, 2019

mbukatov Jun 19, 2019

vasukulkarni commented Jun 18, 2019

clacroix12 commented Jun 18, 2019

petr-balogh commented Jun 18, 2019

clacroix12 commented Jun 18, 2019

mbukatov left a comment •

edited

Loading

mbukatov commented Jun 19, 2019

RazTamir commented Jun 20, 2019

Polarion support #243

Polarion support #243

Conversation

clacroix12 commented Jun 17, 2019

mbukatov left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dahorak Jun 19, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vasukulkarni commented Jun 18, 2019

clacroix12 commented Jun 18, 2019

petr-balogh commented Jun 18, 2019

clacroix12 commented Jun 18, 2019

mbukatov left a comment • edited Loading

Choose a reason for hiding this comment

mbukatov commented Jun 19, 2019

RazTamir commented Jun 20, 2019

mbukatov left a comment •

edited

Loading

dahorak Jun 19, 2019 •

edited

Loading

mbukatov left a comment •

edited

Loading