[Feature] Add meta data to test case

[claytonneal]

Hiya,

It would be good to add key/value meta data to a test case, e.g.
[META id=123, category=X]
The meta data isnt needed to select tests for execution, or needed in the output reports
However the meta data should be able to be accessed with keywords, e.g.
${ID}= Get Test Meta Data id
${CATEGORY}= Get Test Meta Data category

What this will be useful for is something like:
[META testlinkid=123 storyid=321]
And then to be able to access this meta data in the test case teardown, to pass to external libraries to report results in TestLink or even Zephr.

Thanks.
Clayton

[pekkaklarck]

Why cannot you use tags for this purpose? You could use, for example, id=123 tag and then parse that in a simple custom library:

def get_test_id(tags):
    for tag in tags:
        if tag.startswith('id='):
            return tag[3:]
    raise ValueError('No id found!')

*** Test Cases ***
Example
    [Tags]    id=123
    ${id} =    Get Test Id    ${TEST TAGS}

[pekkaklarck]

Adding metadata to tests as name-value pairs has been proposed also earlier. The reason all proposals have been rejected include:

1. Nobody has come up with use cases that couldn't be implemented with tags.
2. It would be a lot of work to implement. It needed to be parsed, added to all internal models, shown in logs and reports, etc.
3. Having two very similar concepts is confusing.

If you have use cases (1.) and are willing to provide a pull request (2.), we can think is the third issue alone severe enough to prevent this to be added.

[claytonneal]

Hello.

The meta data isnt needed for test selection, or reporting.
The idea is to add meta data to a test that can be used within keywords
For example when developing Web UI tests, it would be good to add meta data to a test to say its desktop/mobile or specific browser specific. This data can then be read (via keywords) in the test setup to know how to instanciate the browser it will use.

As meta data isnt aimed to be reported on, Tags arent a good substitute, as they are reported on. So unless there is an option to not report the results of certain tags!?

Cheers.
Clayton

[pekkaklarck]

There are --tagstatinclude and --tagstatexclude to configure tag statistics. Further tag related configuration is possible too if there are good use cases.

[claytonneal]

Actually thinking about it exclude include tags wont work as there will be so many..

Reading a bit more the user guide I see there is possibility to add meta data to test suits. This enhancement is to add the same to test cases and expose via a get keyword.

[pekkaklarck]

It's possible, for example, to exclude all id tags with --tagstatexclude id=* or use --tagstatexclude *=* to exclude all tags that use name=value syntax. Is that enough for you? If not, what would be needed?

Notice that even when tag stats are included/excluded, tests themselves will have those tags. If that's a problem, you could dynamically remove tags using Remove Tags keyword or remove tags from outputs after execution. Personally I'd prefer keeping that information visible, though.

I know that suites have metadata that is defined as name/value pairs and understand uou'd like tests to have them too. I don't think that's a good idea for the following reasons:

• Tags can be used to emulate name/value pairs by using syntax like name-value, name: value or name=value. Enhancements to make this usage easier and better are possible.
• Having two similar features would make learning harder. I don't like new users thinking should they use metadata or tags.
• Adding metadata to tests would be quite a big task.

[pekkaklarck]

I knew this was proposed and rejected before, and searched old issues to see what we have commented earlier. The original issue with most comments is #688, but there's also #1039 as well as #1487 that seems to be submitted by you.

Three individual people have asked this, so there clearly is demand for this feature. After reading the comments to the aforementioned issues, I still think it's better to use tags for this purpose. That's a functionality that already exists and that can be still improved. In my opinion the confusion added by a new feature that does pretty much the same thing isn't worth the possible benefits.

It would be very interesting to see how well tags actually work in this kind of usage, though. I already showed simple code that can parse information from tags in my first comment and proposed using --tagstatexclude *=* in the previous comment. Could you test how well these ideas work in practice? If you encounter problems, we can see could we enhance tags somehow or would separate metadata work better.

Notice that even if we would decide that tests should get metadata in addition to tags, it would still require someone needing it badly enough to implement it. If you think it is a very important feature, you could implement it yourself and compare results against just using tags. If results with metadata are much better and you have working code to implement them, it would be much easier to persuade us to merge a pull request.

[claytonneal]

Ok thanks for the help on using tags... will get back on how it goes! Im not a python dev, so unlikely a merge request will come from me :-) Lets close the ticket!

[pekkaklarck]

Cool, waiting to hear how tags work for you and is there something that we could enhance in them!

Here's variant of the original library that parses the metadata from tags. This time it gets tags automatically from Robot, requires giving the name of the metadata you want as an argument, and handles matadata names case-insensitively:

from robot.libraries.BuiltIn import BuiltIn

def get_test_metadata(name):
    tags = BuiltIn().get_variable_value('${TEST TAGS}')
    prefix = name.lower() + '='
    for tag in tags:
        if tag.lower().startswith(prefix):
            return tag[len(prefix):]
    raise ValueError("Metadata '%s' not found!" % name)

Assuming you put that code to a file named TestMetadata.py, you can use it like this:

*** Settings ***
Library    TestMetadata.py

*** Test Cases ***
Example
    [Tags]    id=42    category=foo
    ${id} =    Get Test Metadata    ID
    ${cat} =    Get Test Metadata    category
    Log Many    ${id}    ${cat}