Parallel Tests in Multiple Browser Are Indistinguishable on TeamCity

[LukeKeller]

If you run your tests in parallel in different browsers, multiple tests of the same name show up. For example, with webdriverIO you can define multiple browsers in your config file to run a suite across multiple browsers. Unless you modify the name of the test in the reporter to explicitly state what browser it is run on, you will get two tests with the same name.
If a test fails in one browser, it is more difficult to determine which browser had an issue. You are also not able to easily compare how tests are performing in different browsers.
My team wrote our own TeamCity reporter that we have been using for a few weeks now and our solution was to append the browserName to the test name in the reporter.
Something along the lines of test.runner[Object.getOwnPropertyNames(test.runner)].browserName should get you the name of the browser.

It also looks like the tests get nested inside each other during parallel execution which may mess up timings.

[sullenor]

I haven't tried to run tests in multiple sessions in teamcity yet, so I haven't seen possible issues here. I'll checks your solution.

Thank you for detailed the report.

[sullenor]

@AdmiralAckbar142 I wonder if the sessionId will be better instead of the browser name. What do you think?

[LukeKeller]

@sullenor
We chose to use browser name because we are running these tests in parallel across chrome, ie and firefox. So having the browser name made it easy for us to distinguish them on TeamCity reports. When looking at the results in TeamCity, I'm not sure that the sessionId will be of much use.

We also never added in logic to only use this when tests are in parallel, so even our suites that only execute in a single browser will append the browser name to the test name. If I have time I may play around with that. I think test.runner will be an array if they are in parallel, so I think that could be used as a check.

[sullenor]

I see your point for the browser name. I thought about case when only one browser will be used. In that case browser name will not be uniq.

Anyways, according to the teamcity documentation it is possible to identify messages using flowId attribute:

Message FlowId
The flowId is a unique identifier of the messages flow in a build. Flow tracking is necessary for example to distinguish separate processes running in parallel. The identifier is a string that should be unique in the scope of individual build.
##teamcity[ flowId='flowId' ...]

So, providing it should help here.

Talking about message format, what do you think about adding possibility to provide the custom pattern for the messages? So you'll be able specify fields that you want to see in the report. May be smthing similar to https://github.com/webpack/css-loader#local-scope from webpack css-loader

[sullenor]

I published fix for the usage in parallel and I'll keep it open to add support for the custom message format. So you'll be able to specify smthing like this:

exports.config = {
  // ...
  reporters: ['teamcity'],
  reporterOptions: {
    message: '[browser] [title]',
  },
  // ...
};

Also it will be possible to specify the filename output.

[LukeKeller]

I think I'm still seeing some nesting issues. I've tried your updated version and incorporating flowId into our version of TC reporter, but it still seems off. For example, running a set of smoke tests that contains a single test that executes in chrome and ie in parallel. Both suites and testNames get browser appended to end to give them independent names. Whichever one starts second gets both suite names and the first one has a blank suite name in team city.
It looks like it is happening because the second one starts before the first has finished and TC nests it inside the first one in the build log. Based on their documentation, it looks like flowId should resolve this, but doesn't appear to be.
I will continue to look into this next week.

[sullenor]

@AdmiralAckbar142 thank you for the information. I'll try to look for more solutions.

[sullenor]

@AdmiralAckbar142 I have a wild guess. Looks like providing sessionId instead of specHash to flowId property should help to solve that issue. Earlier I used specHash which is generated from filepaths according to webdriverio code.

Anyways, I published the new version with possibility to customize messages, so you can distinguish tests using reporterOptions: { message: '[browser] [title]' } in the config.

Up: or may be flowId simply not working.

[kisenka]

@sullenor hello. Cannot find release with sessionId instead of shecHash. Do you publish it?

[sullenor]

@kisenka thank you for pr. Hope it settles things down.
Published in 1.1.1.