polarizer-py is a python library to collect metadata about your tests via decorators which wrap your test functions. The decorators essentially do two things:
- Create or update a mapping.json file (which maps tests to a unique TestCase ID)
- Generates an XML testcase definition file useable by the Polarion TestCase importer
By generating the mapping.json file, as well as the XML test definition file, this allows a team to use the polarizer-vertx webservice to create or update TestCase definitions in Polarion
polarizer-py uses a configuration that will be read in from ~/.polarizer/polarizer-testcase.yml|json. It looks like this:
---
project: RedHatEnterpriseLinux7
author: stoner
mapping: /home/stoner/dummy-map.json
definitions-path: /home/stoner/PycharmProjects/metatest/tests/definitions/metadata.yaml
new-testcase-xml: /home/stoner/testdefinitions.xml
servers:
polarion:
url: https://path.to.polarion.com
user: myteam
password: mypassword
testcase:
endpoint: /import/testcases
timeout: 300000
enabled: false
selector:
name: rhsm_qe
value: testcase_importer
title:
prefix: "RHSM-TC: "
suffix: ""- project: name of your project in Polarion
- author: who (mostly) authored this testcase
- mapping: path to a json file which is used to map testcase name to a Polarion TestCase ID
- definitions-path: path to a yaml file that has all the data needed to define a testcase in Polarion
- new-testcase-xml: path where to write the xml definition file that can be sent to the Polarion TestCase importer
- servers:
- polarion:
- url: the url of the polarion server to communicate with
- user: user name credentials
- password: password for user
- polarion:
- testcase:
- endpoint: the endpoint to import testcases on Polarion
- timeout: how long to wait for response message until considered a failure
- enabled: if false, dont actually make an import request, even if some testcase ID's are empty
- selector:
- name: Name part of a JMS selector string. "{name}='{value}''"
- value: Value part of a JMS selector string as shown above
- title:
- prefix: An optional string that will be prefixed to an autogenerated title (if title is empty, it defaults to name of test method)
- suffix: An optional string that will be appended to an autogenerated title
The mapping json file is how we can map a testcase name to its unique ID. It is used so that we can insert the unique ID into the xunit result file, since we know the name of the method and can look up its ID. If you use data driven tests, it also contains the parameter names used in the function.
A simple mapping.json file looks like this:
{
"metatest.testmeta.test2": {
"RHEL6": {
"id": "RHEL6-12345",
"params": [
"name"
]
}
}
}When you decorate a method with @metadata, it will find the function's corresponding definition in the yaml file and then try to find it in the mapping.json file too. If it doesn't exist in the mapping.json file, an entry will be created for it.
If it already exists in the mapping.json file, a comparison will be done to make sure the files are in agreement. If there is a deviation, an exception will be raised.
If the id field is an empty string, then we know we need to make a Polarion TestCase Import request. Once the response returns, the new TestCase ID will be with it, and the mapping.json and the yaml definition files will be updated.
As a side note, all this data could have been kept in a single file...perhaps the mapping.json file. However, we wanted to keep the mapping.json file small, since it needs to be uploaded to the polarizer service so that if also given a regular xunit file, a Polarion compatible xunit file could be returned.
Ideally, all this information should be stored in a database as a future improvement
polarizer-py either uses YAML files to define the information for a testcase, or a python dict. Ultimately, the yaml file gets loaded as a dictionary, and any testcase subdict whose id == "" gets put in a list for an import request.
Once an import request has been made, and the Polarion TestCase ID is returned in a response message, both the id in the definition file, and the id in the mapping JSON file will be updated.
The decorator also works to ensure that the ID's contained in the definition file and the mapping file are kept in synch, and will warn you if they diverge somehow. Because we can edit files easily (eg the yaml definition file or the mapping.json file), it is preferred to use the definition files instead of a python dict. Eg
# prefer this
@metadata(path="/tmp/mydefinition.yml")
def foo():
pass
# to this:
@metadata(definition={
"project": "RHEL6",
"testcase_id": ""
})
def bar():
passThe default definition file located in a path as determined by the configuration file
---
- testcase:
name: metatest.testmeta.test1
project: RedHatEnterpriseLinux7
title: Defaults to name, but can be anything if empty
id: ""
description: Just a test
# Normally, the test-steps can be left blank, as we can determine this programmatically
#test-steps:
#- test-step:
# - test-step-column:
# - parameter:
# scope: "local"
# name: "foo"
custom-fields:
caseimportance: medium
caseautomation: automated
caselevel: component
caseposneg: positive
casecomponent: ""
testtype: functional
subtype1: ""
subtype2: ""
tags: comma,separated,values
linked-workitems:
- linked-workitem:
- workitem-id: RHEL7-23456
- role-id: verifies
update: false
- testcase:
name: metatest.testmeta.test4
project:
- RedHatEnterpriseLinux7
- RHEL6
title: Defaults to name, but can be anything if empty
id: ""
description: Just a test
custom-fields:
caseimportance: medium
caseautomation: automated
caselevel: component
caseposneg: positive
casecomponent: ""
testtype: functional
subtype1: ""
subtype2: ""
tags: comma,separated,values
linked-workitems:
- linked-workitem:
- workitem-id: RHEL7-23456
- role-id: verifies
update: falseA custom definition file in some other location. The @metadata decorator can use the path keyword argument to point to a custom definition file.
- testcase:
name: metatest.testmeta.test2
project: RedHatEnterpriseLinux7
title: Defaults to name, but can be anything if empty
id: ""
description: Just a test
custom-fields:
caseimportance: medium
caseautomation: automated
caselevel: component
caseposneg: positive
casecomponent: ""
testtype: functional
subtype1: ""
subtype2: ""
tags: comma,separated,values
linked-workitems:
- linked-workitem:
- workitem-id: RHEL7-23456
- role-id: verifies
update: falseThe metadata decorator has 3 possible ways to be used:
- with the path keyword set to a file location
- with nothing at all, in which case it uses the yaml definition file indicated from the configuration
- with the definition keyword set to a python dict
If the first two options are used, the metadata will try to look up the fully qualified name of the function (eg package.module.function name) in the definition file either given by the path keyword argument, or the default file. The metadata decorator will convert the yaml file to a python dict, and if the file doesn't exist, it will raise an exception.
Once the dict is created, the decorator will look up the qualified name
Here's a simple example:
from polarizer_py.metadata import metadata, config
test_dir = config()["definitions-path"] # Path(this_dir, "tests", "definitions")
@metadata(path=test_dir)
def test2(name: str) -> str:
print("you passed in {}".format(name))
return name + ": got it"
@metadata()
def test1():
print("This is just a test")
class MyTest:
def __init__(self, x):
self.x = x
@metadata(definition={
"project": "RHEL6",
"id": "",
"custom-fields": {
"importance": "",
"automation": "",
"caselevel": "",
"caseposneg": "",
"casecomponent": "",
"testtype": "functional",
"subtype1": "",
"subtype2": "",
"tags": "comma,separated,values"
}
})
def test3(self, y, bar=""):
print("Just seeing how it works as a method")
return y + barThe example above shows three different ways to annotate your tests.
The first is just a function, and it uses a custom path to look for the yaml definition. The second uses the default definition file as indicated by the configuration file. The last shows how you can specify a python dict. The main disadvantage with the last method is this is that the source code can not be edited to get the new ID after the import request has been made.
So the author has to manually go back and update the ID
Right now, the project is in very early alpha stage. Some work needs to be done to detect which modules are using polarizer_py.metadata.
- First, import the metadata function into your module(s) and start decorating your test functions.
- Open up a python shell
- Import your modules that imported polarizer_py.metadata
- Once you have imported all modules that use the @metadata decorator:
- from polarizer_py.metadata import MetaData
From there, you can inspect the MetaData class. For example, look at the MetaData.import_list. If it is non-empty, you can generate an XML file
from metatest import testmeta # by importing this module, any functions decorated with @metadata in testmeta will run
from polarizer_py.metadata import MetaData # MetaData class holds the info we need
from pprint import pprint
pprint(MetaData.import_list)
MetaData.make_testcase_xml()There's still a bit of work that needs to be done:
- Import hooks to know what modules imported polarizer_py.metadata
- upload the generated testcase definition xml to polarizer-vertx
- Edit the definition yaml file from the returned mapping.json file
A separate web service will take the XML testcase definition file and the mapping.json file, and return a new mapping.json file with the new TestCase ID
curl -F tcargs=@/home/stoner/polarizer-testcase.json -F mapping=@/home/stoner/Projects/myproject/mapping.json -F tcxml=@/home/stoner/testdefinitions.xml http://localhost:9000/testcase/import