Concept listener start_suite hook isnt correct.

[sawt00th]

Hi
Recently i started to write someone listeners, for data-driven test, and faced with some concept error in hook processing.
First of all let show tests livecycle:

• start_suite
• start_test
• end_test
• end_suite

Obviously, we will see such a picture if we make a minimal listener and use one test.
Thats scheme has problems in any suite setups processing, because start_suite was run before any keyword call, and we havent possibe sets vars inside our listener from suites setups for each suite.
Moreover, tearUp/tearDown functions (Suite Setup/Suite Teardown) directly not related with tests and testing-run time. They are a separate stage, and must stoped all tests in suite and skip their, if somethiing was wrong (for example, we cant conect to DB or open file - why we continue running in that suite?)

Parametrize (templating)
If we look at our log file where a parameterized test was used, we will see that the test actually did not become parameterized, but only began to contain many key-words, and if one or more errors occurred inside the parameters, the report will contain only one mistake for this test, which is essentially not true.
Conclusion (as for me)

1. setup-section is a self-sufficient unit in test-cycle.
2. setup section in log/report must be as separate step
3. Templates must generate test-case for each of parameters, and has chort information about all of generated tests (for example all generated test from template has some tag like "Generate from //templates name//")
   Listener
   `import json
   from robot.api import logger

class My:
ROBOT_LIBRARY_SCOPE = 'GLOBAL'
ROBOT_LISTENER_API_VERSION = 2

def __init__(self):
    self.ROBOT_LIBRARY_LISTENER = self
    self._counter = 0
    self.global_reference_dir = None
    self.suite_reference_dir = None
    self.current_section = None
    self.data = None

def count(self):
    """User words"""
    self._counter+=1


def set_reference_dir(self, name):
    """Settings for suite step, we want set diffrent reference dir, for eatch test suites"""
    self.suite_reference_dir = name
    logger.warn('Reference dir now set: {}'.format(name))

def set_section(self, name):
    """Settings for suite step, we want set other section in data-base file for each suite (work same with set_referense_dir)"""
    self.current_section = name
    logger.warn('Section now set: {}'.format(name))


def _end_keyword(self, name, attr):
    logger.warn('<p>Keyword calls: {}</p>'.format(name))

def _start_suite(self, suite, result):
    """we want check our parameters on start suite"""
    logger.warn('<p>Start Suite with:</br>'
                'current sections - {}</br>'
                'suite reference - {}</br>'
                'global reference - {}</p>'.format(self.current_section, self.suite_reference_dir, self.global_reference_dir), html=True)

def _start_test(self, test, result):
    logger.warn('<p>Start Test with: {} {}</p>'.format(test, result), html=True)

def _end_test(self, name, attrs):
    logger.warn('<p>End Test with: {} {}</p>'.format(name, attrs), html=True)

def _end_suite(self, name, attrs):
    logger.warn('<p>End Suite with: {} {}</p>'.format(name, attrs), html=True)

def _close(self):
    logger.warn('<p>Close:</p>', html=True)`

Testcase:
*** Settings ***
Documentation Suite description
Suite Setup Settings Current Suite

*** Test Case ***
Test One
[Tags] Compare
Count
Count
Count

*** Keywords ***
Settings Current Suite
Set Reference Dir referencedir
Set Section Test

working screen (on screen runed two same test-cases)

[pekkaklarck]

The description is pretty confusing with various different topics. I'm not sure did I understand it fully, but here are some comments related to it:

1. start_suite being called before suite setup is by design and won't change.

2. If a test with template has multiple data sets, they are all executed even if one of them fails. Representing this kind of data-driven tests in logs could be enhanced and we already have Data-driven test cases should be shown better in log and testdocs #1702 about that.

Could you clarify was there something that I missed?

[sawt00th]

1. start_suite being called before suite setup is by design and won't change.

setup_suite, need separate hook (start_suite and something like suite_setup) that are called BEFORE the tests in the suite were built.

Best variant look like:

1. suite_load (call when suite with some one be finded)
2. suite_setup(call if Suite Setup keywords was been defined in that suite)
3. suite_start (call after tests was been collect, like now work)
4. test_start
5. test_end
6. suite_end
7. suite_down(call when Suite Teardown keywords was been defined in that suite)

Fixtures (setup/down, start/end functions and etc) and running collecting process - thats diffrent things and hooks must be different, for their.

[pekkaklarck]

Could you clarify why separate hooks would be needed for setups/teardowns and why existing start/end_keyword hooks couldn't be used? An obvious limitation is that the listener API v3 doesn't yet support these methods, but hopefully we can add them some time in the future.

[sawt00th]

Because the setups/teardowns process is an independent step in the execution of tests, and should not depend on the tests.
Initialization and running of tests should be carried out only after the complete and successful implementation of the setups/teardowns process. If we want have controll each of that steps, we must have needed hooks for this
Okey, let me show an example.
We have listener, who has some global state, like db connection, parameters file or something.
Our listener is hybrid (library+listener) and has some keyword two types: configurate and regular words.
And we has two suites (A, B suites) with same tests, but diffrent parameters, who takes from our global file.
In suites we want to modify our global state (edit section file, or set table names), but this edition must save inside listener not in tests location (listener has self-attr current_section, who save selected section of file for this (current) suite.
On next step, considering current_section we want dinamicly generate tests, who based on all value of current_section from our flobal file.
That a main moment.
We can modify listeners attrs in suite_start hooks, but that edition isnt matter for current suites, because tests are creation early. And all code, who try modify some global state in listener in Settings test-section, will be run only AFTER suite creation -> code will breack (if code try use that current_section variable inside self), or be using previos value Settings value (look at the first screen and Setting section in suite code)

[pekkaklarck]

Are you aware that start/end_keyword is called also with setup and teardown? You can easily check the type of the keyword and do something e.g. when a setup starts. The problem is that v3 listeners don't have these methods at all.

[sawt00th]

You can easily check the type of the keyword and do something e.g. when a setup starts.

Pls, look at screenshot, keyword-fixtures called after suite_start.
There is no way to do some actions before collecting test.

The problem is that v3 listeners don't have these methods at all.

The problem is that different things are merged into one hook. Thats true for 2 and 3 API version.
You cannot add tests based on any suite settings that are stored inside the listener and edit from user-side (suite), because test collect first of all, even faster suite settings.

For exaple:
Nose plugin interface has more describe and right way about plugin hook around test process

[pekkaklarck]

Are you saying start_keyword isn't called correctly? Or that you cannot do what you want when it's called? Adding new methods wouldn't fix the latter issue. In general it would be good to know what is your actual problem.