Update reporter to be compatible with wdio v5 #14

tagoro9 · 2019-03-11T15:25:16Z

This PR updates the reporter to be compatible with wdio v5 following the documentation for creating a custom reporter.

I added the same babel configuration as wdio is using and updated the engines to be at least the same that wdio declares.

The main changes in the reporter are:

Turning it into a class
Stop using event emitter in favor of the method for each event
Use write instead of console.log

Fixes #12

The reporter is now a class, so it needs to be correctly transpiled for node

Use same engine as wdio

So classes are transpiled correctly

makepanic · 2019-03-18T14:48:04Z

index.js

  }
 }
+
+module.exports = WdioTeamcityReporter;
+module.exports.reporterName = 'teamcity';


It looks like wdio tries to use this as a ES6 module.
Adding module.exports.default = WdioTeamcityReporter; made it not crash for me, when trying to consume this.

I just added that. It should be working now. I was using importing the reporter instead of specifying the name so I didn't catch that. Thanks!

marr · 2019-03-21T17:39:20Z

bump

kashinross · 2019-03-21T23:38:27Z

I've been testing with this branch in TeamCity and I'm pleased to say it works but I am seeing some issues:

The test count (total number of tests being shown as executed) is inconsistent from run to run
Some tests are being attributed to the wrong suites.
Some suites are being nested under other suites.

Have you come across any of these issues while running your test suites on TeamCity?

tagoro9 · 2019-03-22T23:18:50Z

@kashinross I think this could be because the plugin is using this.write instead of console.log. I'm gonna update the code and see if it is still happening

kashinross · 2019-03-26T18:02:11Z

@tagoro9 I gave it a shot with the latest commit and still seeing similar issues - it is behaving slightly differently before but the issues with suites being combined is still happening. Thanks for looking into it!

christian-bromann · 2019-03-28T22:57:39Z

A v5 reporter should use this.write instead of console.log. Is any support needed?

tagoro9 · 2019-03-28T23:02:39Z

@christian-bromann That is what I was using initially, but I switched to console.log in 9ddfa83 to see if that fixed @kashinross issues, but since that didn't work I'll go back to this.write

christian-bromann · 2019-03-28T23:04:24Z

Please let me know if anything is unclear!

tagoro9 · 2019-04-02T16:20:22Z

I just dropped the last commit that used console.log instead of this.write. @kashinross Let me know if it is still not working for you as expected. It would also be helpful to see the output that you are getting and the expected output.

kashinross · 2019-04-05T20:47:35Z

@tagoro9 Here's a small sample of what's happening

The individual tests are titled as Should do something while the test suites are prefixed with Anon or Authed. Somehow the tests and suites are being nested incorrectly - you can see that the main category is a test that contains multiple other tests and suites nested under it. My tests are organized so that there is only one suite per spec file that may contain multiple tests. There is also an issue where the total number of passing tests changes from run to run. Hope this helps narrow it down!

erinev · 2019-04-26T15:36:24Z

hi there, what is the status of this pull request ? we are onboarding webdriver-io framework in our company and we need TC reporter support (because currently it just shows that build passed, which is inconvinient, but livable ofcourse).
any ETA of merge/release ?

xemicalas · 2019-04-29T10:05:12Z

Hi, any updates when this will be merged?

tagoro9 · 2019-04-29T15:14:40Z

@kashinross Sorry that I didn't reply earlier, it's been some busy weeks. I think can try to dig deeper into your issue this week and see if we can get this to work. (cc @erinev @xemicalas)

kashinross · 2019-06-17T13:12:31Z

@tagoro9 any luck with that?

AndyBarnettCTM · 2019-07-10T09:33:50Z

please merge this lads my build log is so messy and I'm having to use custom failure conditions :'(

dylanlive

Thanks for your work on getting this done!

Regressions

FlowId is no longer being set
Stdout is no longer written on a new line, so it's a giant blob of text

Details

I looked into @kashinross's reported bug. It's caused by parallel tests running.

After investigation, flowID is not being set. This is very important for parallel tests, per teamcity documentation. I've made an in-line pr comment to fix.

It's easily replicable having maxInstances set to 2, and using specs:
spec1.js

describe('scenario 1 file1', () => {
  it('sc1 1', () => {
    browser.pause(500)
  });

  it('sc1 2', () => {
    browser.pause(500)
  });
});

describe('scenario 2 file1', () => {
  it('sc2 1', () => {
    browser.pause(500)
  });

  it('sc2 2', () => {
    browser.pause(500)
  });
});

spec2.js:

describe('scenario 3 file2', () => {
  it('sc3 1', () => {
    browser.pause(500)
  });

  it('sc3 2', () => {
    browser.pause(500)
  });
});

wdio v4 teamcity reporter:
##teamcity[testSuiteStarted flowId='5a9958b1-9ebc-f748-a982-1f055bed574a' name='scenario 1 file 1']

wdio v5 teamcity reporter:
##teamcity[testSuiteStarted flowId='' name='scenario 1 file 1']

In addition, I found while the v4 formatter puts each ##teamcity on a new line, the v5 reporter does not. Made a suggestion in the PR comments.

dylanlive · 2019-07-25T06:08:23Z

lib/message.js

@@ -20,20 +20,20 @@ const { camelCase, get, invoke } = require('lodash');

 const paths = {
  browser: ({ suite }) => ['suite', 'runner', suite.cid, 'browserName'],
-  details: () => 'suite.err.stack',
+  details: () => 'suite.error.stack',
  flowId: ({ suite }) => ['reporter', 'baseReporter', 'stats', 'runners', suite.cid, 'sessionID'],


this needs to be updated. I fixed using:

flowId: () => 'reporter.browser.sessionId',

This may also work, based on the console.log output of the ctx object, but I haven't tested

flowId: () => 'reporter.driver.sessionId',

dylanlive · 2019-07-25T06:39:08Z

index.js

+      this[event] = (...args) =>
+        flow(buildFormatter(event, opts), msg => {
+          if (msg) {
+            this.write(msg);


There's no new line appended to this.write, so everyone gets dumped as a giant blob with no new lines.

Can you add one via:
this.write(`${msg}\n`)

See example of usage.

tagoro9 · 2019-07-25T15:30:32Z

So sorry about my delay for looking back at this

Thanks for your great feedback @dylanlive! I just pushed some changes addressing it

dylanlive

Thanks for the quick update! This is looking much better. While I haven't ran this in TeamCity, looking at the output looks promising

dylanlive · 2019-07-25T18:25:38Z

lib/message.js

@@ -20,20 +20,20 @@ const { camelCase, get, invoke } = require('lodash');

 const paths = {
  browser: ({ suite }) => ['suite', 'runner', suite.cid, 'browserName'],


I think this might need an update as well. I did a console.log(ctx) to see what object properties are available - you can see the result here
https://gist.github.com/dylanlive/f508a8201f36353954d7e3b03800b297#file-ctx_object-js-L39

Based on that, this works (at least for chrome):
browser: () => 'reporter.browser.capabilities.browserName',

kashinross · 2019-11-13T18:46:30Z

With the latest version of WDIO5 packages ^5.16.0 this seems to be working as expected for me. @tagoro9 what needs to be done to get this released?

tagoro9 · 2019-11-13T18:48:58Z

@kashinross I guess either @sullenor or a contributor to the repo would have to merge this PR and release a new version of the package.

Actual PR: webdriverio-community#14 PR created by: https://github.com/tagoro9

sullenor · 2019-12-14T08:46:15Z

Thank you for your work. I published the 2.0.0 version with this change.

tagoro9 added 7 commits March 6, 2019 16:28

chore(dependencies): add dependencies for wdio 5

f7a4a31

chore(dependencies): add babel-preset-env

2298e16

The reporter is now a class, so it needs to be correctly transpiled for node

chore(engines): update node version

5030557

Use same engine as wdio

chore(babel): use babel-preset-env

ded854a

So classes are transpiled correctly

feat(messages): update event names to new api

cd4d97f

feat(reporter): use new reporter API

858fc60

test(reporter): update tests

b285f96

tagoro9 mentioned this pull request Mar 11, 2019

Make reporter compatible with WebdriverIO v5 #12

Closed

chore(travis): update node versions

1e541b6

makepanic reviewed Mar 18, 2019

View reviewed changes

tagoro9 added 2 commits March 18, 2019 11:41

chore(reporter): export as default

4928dc8

fix(message): use correct path to get error

3cd81af

tagoro9 force-pushed the webdriver_v5 branch from 7932cad to 3cd81af Compare March 18, 2019 18:42

tagoro9 force-pushed the webdriver_v5 branch from 9ddfa83 to 3cd81af Compare April 2, 2019 16:18

dylanlive suggested changes Jul 25, 2019

View reviewed changes

fix(message): use sessionId as flowId

aaa5bab

fix: add new line after every write

4acffa6

dylanlive reviewed Jul 25, 2019

View reviewed changes

fix(message): read browser name from capabilities

aae35f8

dinesh-it added a commit to dinesh-it/wdio-teamcity-reporter that referenced this pull request Nov 18, 2019

Update to support wdio v5 - Patched PR on actual repo

947a9a7

Actual PR: webdriverio-community#14 PR created by: https://github.com/tagoro9

sullenor merged commit 09dc031 into webdriverio-community:master Dec 14, 2019

tagoro9 deleted the webdriver_v5 branch December 16, 2019 15:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update reporter to be compatible with wdio v5 #14

Update reporter to be compatible with wdio v5 #14

tagoro9 commented Mar 11, 2019

makepanic Mar 18, 2019

tagoro9 Mar 18, 2019

marr commented Mar 21, 2019

kashinross commented Mar 21, 2019 •

edited

Loading

tagoro9 commented Mar 22, 2019

kashinross commented Mar 26, 2019

christian-bromann commented Mar 28, 2019

tagoro9 commented Mar 28, 2019

christian-bromann commented Mar 28, 2019

tagoro9 commented Apr 2, 2019

kashinross commented Apr 5, 2019 •

edited

Loading

erinev commented Apr 26, 2019 •

edited

Loading

xemicalas commented Apr 29, 2019

tagoro9 commented Apr 29, 2019

kashinross commented Jun 17, 2019

AndyBarnettCTM commented Jul 10, 2019

dylanlive left a comment

dylanlive Jul 25, 2019

dylanlive Jul 25, 2019 •

edited

Loading

tagoro9 commented Jul 25, 2019

dylanlive left a comment

dylanlive Jul 25, 2019

kashinross commented Nov 13, 2019

tagoro9 commented Nov 13, 2019

sullenor commented Dec 14, 2019

		@@ -20,20 +20,20 @@ const { camelCase, get, invoke } = require('lodash');

		const paths = {
		browser: ({ suite }) => ['suite', 'runner', suite.cid, 'browserName'],

Update reporter to be compatible with wdio v5 #14

Update reporter to be compatible with wdio v5 #14

Conversation

tagoro9 commented Mar 11, 2019

makepanic Mar 18, 2019

Choose a reason for hiding this comment

tagoro9 Mar 18, 2019

Choose a reason for hiding this comment

marr commented Mar 21, 2019

kashinross commented Mar 21, 2019 • edited Loading

tagoro9 commented Mar 22, 2019

kashinross commented Mar 26, 2019

christian-bromann commented Mar 28, 2019

tagoro9 commented Mar 28, 2019

christian-bromann commented Mar 28, 2019

tagoro9 commented Apr 2, 2019

kashinross commented Apr 5, 2019 • edited Loading

erinev commented Apr 26, 2019 • edited Loading

xemicalas commented Apr 29, 2019

tagoro9 commented Apr 29, 2019

kashinross commented Jun 17, 2019

AndyBarnettCTM commented Jul 10, 2019

dylanlive left a comment

Choose a reason for hiding this comment

Regressions

Details

dylanlive Jul 25, 2019

Choose a reason for hiding this comment

dylanlive Jul 25, 2019 • edited Loading

Choose a reason for hiding this comment

tagoro9 commented Jul 25, 2019

dylanlive left a comment

Choose a reason for hiding this comment

dylanlive Jul 25, 2019

Choose a reason for hiding this comment

kashinross commented Nov 13, 2019

tagoro9 commented Nov 13, 2019

sullenor commented Dec 14, 2019

kashinross commented Mar 21, 2019 •

edited

Loading

kashinross commented Apr 5, 2019 •

edited

Loading

erinev commented Apr 26, 2019 •

edited

Loading

dylanlive Jul 25, 2019 •

edited

Loading