Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support more general merging results with Rebot #1687

Closed
spooning opened this issue Jun 30, 2014 · 14 comments

Comments

@spooning
Copy link
Contributor

commented Jun 30, 2014

Originally submitted to Google Code by filip.noetzel on 24 Mar 2014

Hey,

first, some background:

We're using RF to run selenium with ~100.000 test cases across 6 browsers. Obviously, that would take a while if we were to execute them in series. To speed things up, we're using Jenkin's matrix build ability to run test 1-1000 in on job, 1001-2000 in the second job, etc. After all individual jobs have run, we'd like to generate a report that looks as if the tests were executed in series (i.e. parallelisation happens transparently).

second, here's a reduced test case 'split_testcases.robot':

*** Testcases ***

Testcase A
Log A

Testcase B
Log B

When run like

$> pybot split_testcases.robot

one gets a report that has the following hierarchy:

Split Testcases

  • Testcase A
  • Testcase B

However, when run like this:

$> mkdir run{1,2}
$> pybot --outputdir run1 --test 'Testcase A' split_testcases.robot
$> pybot --outputdir run2 --test 'Testcase B' split_testcases.robot

.. and combined with

$> rebot run{1,2}/output.xml

i get a report that has this hierarchy:

Split Testcases & Split Testcases

  • Split Testcases
    • Testcase A
  • Split Testcases
    • Testcase B

I tried using --rerunmerge, but that fails because the test reports contain disjoint testcases, am I missing a command line flag ?

I think I was able to adapt merger.py posted in https://code.google.com/p/robotframework/issues/detail?id=1615 to create a merged result that resembles my wanted output, simply by removing the check. I think I can work with that for now. However, such functionality sounds useful to other RF users.

Thanks again for Robot Framework :)

@spooning

This comment has been minimized.

Copy link
Contributor Author

commented Jun 30, 2014

Originally submitted to Google Code by @pekkaklarck on 14 Apr 2014

Your use case looks totally valid. You are also correct that --rerunmerge doesn't work here. It was designed to handle only a certain merge operation, not to be a generic merger. Adding more merge possibilities is definitely possible as long as they are generic and have somewhat clean usage/semantics.

If I got it right, in your situation tests in merged outputs should be placed into the suite in the original output into corresponding places. In general that is easy, but there are at least the following questions to be answered:

  1. What to do if there already exists a test with same name? Merge them like with --rerunmerge, add new test anyway (there shouldn't be technical restrictions for this), or give an error?

  2. What to do a test in merged output has no corresponding suite? Should we copy the whole suite from the merged output or should this be an error?

  3. If we allow adding also missing suites (see above), what should we do if the top level suites are different?

@pekkaklarck

This comment has been minimized.

Copy link
Member

commented Aug 21, 2014

@peritus, do you have any comments to the above three questions? I think this would be a good feature to add, and it would be good to discuss with some real users about these corner cases. This shouldn't cause any backward-incompatibility issues so we could include it even in RF 2.8.6.

@pekkaklarck pekkaklarck added enhancement and removed bug labels Aug 21, 2014

@mkorpela

This comment has been minimized.

Copy link
Member

commented Aug 23, 2014

I've implemented this feature in https://github.com/mkorpela/pabot

It is not enough to just remove the check - there will be problems when there are subsuites. I'll publish the relevant source code with Apache 2.0 license so that it can be used as a reference implementation for an implementation in RF.

@mkorpela

This comment has been minimized.

@pekkaklarck

This comment has been minimized.

Copy link
Member

commented Aug 23, 2014

Thanks @mkorpela for sharing pabot code. It doesn't answer the question how to merge individual tests, though. I'd like this to work somehow sanely:

pybot --include smoke --output smoke.xml tests
pybot --exclude smoke --output other.xml tests
rebot --parallelmerge smoke.xml other.xml
@mkorpela

This comment has been minimized.

Copy link
Member

commented Aug 23, 2014

Use the same logic on test level that I've just used on suite level?

@pekkaklarck

This comment has been minimized.

Copy link
Member

commented Aug 25, 2014

After thinking about this a bit more and discussing with @jussimalinen, here's our current proposal:

  1. Change the current rerun-merge functionality so that suites and tests in the merged outputs are added to the original suite.
  2. Keep merging same test in same suite together exactly like nowadays.
  3. Having different top-level suites should still be an error.
  4. Add new --merge option that provides this functionality. Keep --rerunmerge as an alias but deprecate it in RF 2.9 and remove later.
  5. Later add some config option to control how to handle same test in same suite if there is a need. The default action should be merge, but keeping tests separate could be supported too and perhaps we could also have strict option to make this an error. If such an option is added, we can either add separate option like --notestmerge or add more generic --mergeconfig that could get also other merge specific sub options.

The above approach shouldn't change the old rerun-merge functionality other than --rerunmerge option being gracefully renamed to --merge. In other words, this should work as expected:

pybot --output original.xml tests/
pybot --rerunfailed original.xml --output rerun.xml tests/
rebot --merge original.xml rerun.xml

This should also support more generic merge nicely. In other words, this example I presented already earlier should work:

pybot --include smoke --output smoke.xml tests
pybot --exclude smoke --output other.xml tests
rebot --merge smoke.xml other.xml

@pekkaklarck pekkaklarck removed the pending label Aug 26, 2014

@pekkaklarck pekkaklarck modified the milestones: 2.9, 2.8.6 Aug 26, 2014

@pekkaklarck

This comment has been minimized.

Copy link
Member

commented Oct 3, 2014

This was implemented by revision e8d3f7c. As noted in the commit message, these tasks are still TODO:

  • Decided how to add a note that tests/suites have been added/replaced. Currently test/suite message is updated, but could also use documentation and with suites also Metadata.
  • Documentation in User Guide.
@pekkaklarck

This comment has been minimized.

Copy link
Member

commented Oct 3, 2014

Revision a5c85e5 renamed rerunmerger.ReRunMerger -> merger.Merger.

@simplyviks

This comment has been minimized.

Copy link

commented Aug 30, 2016

Hi,

i have a requirement where i need to break a single testcase in multiple in report.xml.

i have a testcases A(Scenarios A)under suit A.
now in this testcase A i am covering many more testcases say A1,A2..

but in the report as expected the testcase count is 1.

Expected report.xml in my case.

for example.
total test case:3(A+A1+A2)
passed:2
failed:1
how do i achieve this??

do i need to write a custom result writer and rebot script??

any leads is appreciated,Thanks in advance.

@glgv

This comment has been minimized.

Copy link

commented Oct 18, 2016

Hi,

I am also having the same requirement as 'simplyviks' posted here. I need to generate multiple logs by splitting the log file from one testcase run.

Any pointers are greatly appreciated. Thank you.

@simplyviks

This comment has been minimized.

Copy link

commented Oct 19, 2016

I tweaked the counting logic.though that's not the best solution.

@mkorpela

This comment has been minimized.

Copy link
Member

commented Oct 20, 2016

I think you should use tags for reporting purposes (requiremetents etc.) instead of counting test cases.
You can cover multiple tags with one test case and dynamically add and remove tags during test execution. Use "BuiltIn.Set Tags" and "BuiltIn.Remove Tags".

@glgv

This comment has been minimized.

Copy link

commented Oct 20, 2016

Hi, Thanks for sharing your comments.

@simplyviks, can you share how you tweaked the counting. In my case i'm using dummy testcases for getting the count and pass/fail result addressed. But for that dummy testcase, i also need to associate the respective log which is a snippet of the full log run in one major testcase. Are you also getting the log seperately?

@mkorpela, I am able to set/remove tags for sections of codes under same testcase. But i am unable to get seperate log files based on the tags. I tried rebot with different options like '--tagstatexclude', '--removekeywords tag:' but with no luck. Is there a way i can extract logs for only the code part which is under defined TAG?

Thanks again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.