Easily creating testruns from inputdata

[JockeJarre]

I am trying to produce automated test runs with different input parameters using pabot.
I found out, by using the argumentfile[0-9] parameter, I can let pabot run the same robot file with the different parameter files read in from the command line.
I wanted to have a more automated function for this, so I created a POC in the fork JockeJarre/pabot.
Now you can specify an argumenttemplate and a csv file with input data.

argumenttemplate.txt
--variable reportname:${reportname} --variable othervalue:${othervalue}

argumentdata.csv
reportname;othervalue\n firstname;othervalue1\n nextname;othervalue2

argumenttemplate.robot
*** Settings ***
*** Variables ***
${reportname} testname
${othervalue} othervalue

*** Test Case ***
Testing argumenttemplate
Log Do something with the values (for example run different reports) ${reportname} ${othervalue}

Command line:
pabotdevel.bat --argumenttemplate argumenttemplate.txt --argumentdata argumentdata.csv --name ArgumentExample argumenttemplate.robot

When run, the CSV file is read, and using the template, it automatically produces many argumentfiles and automatically run all these using the existing functionality.

I do not know if there is a better more simple way of producing the same thing?

[mkorpela]

Hi @JockeJarre,
Sorry for my late reply.
Your approach seems interesting! It has better comparability for different configurations as they are all located at the same file.
Have to think if there is a way to integrate this functionality to pabot.

[JockeJarre]

Hi,
no problem.

Good you like the idea. I just tested it under py3... sorry for breaking py3 support...
Will improve it soon.
First iteration to also make my addition py3 compatible done.

[Noordsestern]

The excellent datadriver does generate test cases from csv data: https://github.com/Snooz82/robotframework-datadriver

Maybe there is a neat setup to run those data driven tests in parallel with pabot.

[JockeJarre]

The excellent datadriver does generate test cases from csv data: https://github.com/Snooz82/robotframework-datadriver

Maybe there is a neat setup to run those data driven tests in parallel with pabot.

I tested robotframework-datadriver, but I do not get it to work together with pabot. The reason is that pabot starts the robot instances with the --Test "Test Name" parameter and robot cannot find the automatically generated tests at this "early stage".

I also tested to rewrite robotframework-datadriver to use the --prerunmodifier functionality, but still it is not producing the new tests before the --test parameter kicks in.

for this "problem" in robot framework I opened an issue there: robotframework/robotframework#3265

But I really liked the concepts of the robotframework-datadriver, especially when you just have to use it as a library and it dynamically generates your tests with correct names and so on. Pabot does actually understand that there are tests made.

[Snooz82]

Hi Guys,

Thanks for the flowers!
I really like that somebody uses ist.

Lets try to bring Datadriver and Pabot together.

That i understand what is needed.
You want to make a TestLevel split?
Parallel exec of tests of a single suite?

Or is the issue with datadriver and Pabot, that pabot needs to know the ammount of tests beforehand

[JockeJarre]

Hi Guys,

Thanks for the flowers!
I really like that somebody uses ist.

Lets try to bring Datadriver and Pabot together.

That i understand what is needed.
You want to make a TestLevel split?
Parallel exec of tests of a single suite?

Or is the issue with datadriver and Pabot, that pabot needs to know the ammount of tests beforehand

Hi, yes I liked your Robot plugin, I describe something similar in this enhancement proposal against pabot. With your code we would not need the enhancement I did...

Yes I want to make a TestLevel split, but here robot itself has a problem with the automatically generated tests and the --test parameter correctly generated by pabot. When --test is used it tries to find the named test which does not yet exist.

So I beleive the problem is really with robot and I therefore created the issue described above against robot instead.

[JockeJarre]

hi,

@pekkaklarck came back with his thoughts on why and if to change the -test flag on robot:
robotframework/robotframework#3265 (comment)
I also mentioned our discussions to solve it between pabot and datadriver instead.

[karthikcat]

Hi All ,

We are running the robotframework testsuite in a seperate run template file.py , by importing robot and run function in it .
run(TESTDIR,
outputdir=OUTPUTDIR,
output=OUTPUTFILE,
log=OUTPUTLOG,
report=OUTPUTREPORT,
loglevel=OUTPUTLOGLEVEL,
variable=['ENV:' + ENV,
'BROWSER:' + BROWSER,
'ASSET:' + ASSET,
'DEVICE_TYPE:' + DEVICE_TYPE],
variablefile=[VARIABLES_FILE, PASSWORD_FILE_PATH],
include=INCLUDE_TAGS,
exclude=EXCLUDE_TAGS,
reporttitle=REPORT_TITLE,
logtitle=REPORT_TITLE,
tagstatinclude=INCLUDE_TAGS,
tagstatexclude=EXCLUDE_TAGS,
tagstatcombine=INCLUDE_TAGS_COMBINED,
metadata=METADATA)

Can we do parallel execution with pabot in this seperate run template file . If so can you please share me the code or the logic where to add this pabot in the run function of robot module

Thanks in Advance !!!!!

[karthikcat]

Hi All ,

Example::

    from robot import run

    run('path/to/tests.robot')
    run('tests.robot', include=['tag1', 'tag2'], splitlog=True)
    with open('stdout.txt', 'w') as stdout:
        run('t1.robot', 't2.robot', name='Example', log=None, stdout=stdout)

Equivalent command line usage::

    robot path/to/tests.robot
    robot --include tag1 --include tag2 --splitlog tests.robot
    robot --name Example --log NONE t1.robot t2.robot > stdout.txt
"""

can anyone suggest for pabot ????

[JockeJarre]

Well, pabot is meant to be started on the command line shell or cmd in windows.
There it then will spawn many processes in parallel and collect the result afterwards. So I do not think that you will get very happy with how you start it currently. Maybe it could be possible to do """ from pabot import run""" something, but that I have no clue about.

What I am mentioning here was an idea how to make pabot able to read input data and run the tests with that data in parallel, but there is already a nice standard plugin for robot doing this already called https://github.com/Snooz82/robotframework-datadriver by @Snooz82 (René). So we are talking how to make pabot and datadriver work better together.

Please remember this is all work done voluntary, so you will get the answer when it comes.

[JockeJarre]

Hi Guys,
Thanks for the flowers!
I really like that somebody uses ist.
Lets try to bring Datadriver and Pabot together.
That i understand what is needed.
You want to make a TestLevel split?
Parallel exec of tests of a single suite?
Or is the issue with datadriver and Pabot, that pabot needs to know the ammount of tests beforehand

Hi, yes I liked your Robot plugin, I describe something similar in this enhancement proposal against pabot. With your code we would not need the enhancement I did...

Yes I want to make a TestLevel split, but here robot itself has a problem with the automatically generated tests and the --test parameter correctly generated by pabot. When --test is used it tries to find the named test which does not yet exist.

So I beleive the problem is really with robot and I therefore created the issue described above against robot instead.

Hello, I have tested your changes in pabot @mkorpela and datadriver @Snooz82.

I guess we can close this issue as solved. We have a much more flexible solution than the POC I created!

Thanks for your support both!

/Richard

[Snooz82]

Yes. Lets close it.

[mkorpela]

Ok. Closing in 5, 4, 3, 2, 1