How do I get my results to show up in tests tab?

[codewithtyler]

Currently, I only get results in the Output section of my VSTS build. It'd be great if I could get the results to appear in the Tests section like they do when I run my unit tests. Any advice on how I can set this up would be much appreciated!

[carlowahlstedt]

Off the top of my head, I don't know exactly the best way to integrate into the tests section. I know there are extension points, but I'm not sure if the output can be specified in such a way that it knows to go there. I'll mark this as needs research, but please feel free to comment back here if you find anything more out.

[sebcaps]

@tylerbhughes to do so we just output result as Junit file ( reporters option = junit) and then added a publish result task.

[codewithtyler]

@sebcaps I tried using junit, html, and json in the Reporters setting. Each one generated an error when trying to use it. Are there any other settings I should set in addition to that one?

[sebcaps]

@tylerbhughes can you please share logs ? (ie line starting with [command] and error below)
The only 'issue' we encountered is usage of '. You don't need it in config.
We set cli,junit and worked like a charm

[codewithtyler]

Here's the log from each run. Since this is for work I've removed repo name, and repo URL.

json reporter

2018-02-08T14:58:55.3022476Z [command]C:\Windows\system32\cmd.exe /D /S /C "C:\npm\newman.cmd run --ssl-client-cert C:\agent\_work\28\s --ssl-client-key C:\agent\_work\28\s --reporter-html-template C:\agent\_work\28\s -r json -n 1 -e "C:\agent\_work\28\s\tests\postman\environment.json" "C:\agent\_work\28\s\tests\postman\collection.json""
2018-02-08T14:59:05.2141369Z ##[error]C:\npm\newman.cmd failed with return code: 1
2018-02-08T14:59:05.2141369Z ##[section]Finishing: Newman - Postman
2018-02-08T14:59:05.2141369Z ##[section]Starting: Post Job Cleanup
2018-02-08T14:59:05.2297635Z Cleaning any cached credential from repository: <RepoName> (Git)
2018-02-08T14:59:05.2297635Z ##[command]git remote set-url origin <RepoURL>
2018-02-08T14:59:05.2610128Z ##[command]git remote set-url --push origin <RepoURL>
2018-02-08T14:59:05.2766357Z ##[section]Finishing: Post Job Cleanup
2018-02-08T14:59:05.2922632Z ##[section]Finishing: Run Postman Tests

junit repoter

2018-02-08T14:50:39.0175467Z [command]C:\Windows\system32\cmd.exe /D /S /C "C:\npm\newman.cmd run --ssl-client-cert C:\agent\_work\28\s --ssl-client-key C:\agent\_work\28\s --reporter-html-template C:\agent\_work\28\s -r junit -n 1 -e "C:\agent\_work\28\s\tests\postman\environment.json" "C:\agent\_work\28\s\tests\postman\collection.json""
2018-02-08T14:50:45.7677704Z ##[error]C:\npm\newman.cmd failed with return code: 1
2018-02-08T14:50:45.7677704Z ##[section]Finishing: Newman - Postman
2018-02-08T14:50:45.7677704Z ##[section]Starting: Post Job Cleanup
2018-02-08T14:50:45.7833942Z Cleaning any cached credential from repository: <RepoName> (Git)
2018-02-08T14:50:45.7833942Z ##[command]git remote set-url origin <RepoURL>
2018-02-08T14:50:45.7990196Z ##[command]git remote set-url --push origin <RepoURL>
2018-02-08T14:50:45.8302709Z ##[section]Finishing: Post Job Cleanup
2018-02-08T14:50:45.8302709Z ##[section]Finishing: Run Postman Tests

html reporter

2018-02-08T14:48:40.9494515Z [command]C:\Windows\system32\cmd.exe /D /S /C "C:\npm\newman.cmd run --ssl-client-cert C:\agent\_work\28\s --ssl-client-key C:\agent\_work\28\s --reporter-html-template C:\agent\_work\28\s --reporter-html-export results.html -r html -n 1 -e "C:\agent\_work\28\s\tests\postman\environment.json" "C:\agent\_work\28\s\tests\postman\collection.json""
2018-02-08T14:48:50.8433486Z { Error: EISDIR: illegal operation on a directory, read
2018-02-08T14:48:50.8433486Z     at Object.fs.readSync (fs.js:675:18)
2018-02-08T14:48:50.8433486Z     at tryReadSync (fs.js:540:20)
2018-02-08T14:48:50.8433486Z     at Object.fs.readFileSync (fs.js:583:19)
2018-02-08T14:48:50.8433486Z     at new PostmanHTMLReporter (C:\npm\node_modules\newman\lib\reporters\html\index.js:53:42)
2018-02-08T14:48:50.8433486Z     at C:\npm\node_modules\newman\lib\run\index.js:319:68
2018-02-08T14:48:50.8433486Z     at arrayEach (C:\npm\node_modules\newman\node_modules\lodash\lodash.js:537:11)
2018-02-08T14:48:50.8433486Z     at Function.forEach (C:\npm\node_modules\newman\node_modules\lodash\lodash.js:9344:14)
2018-02-08T14:48:50.8433486Z     at C:\npm\node_modules\newman\lib\run\index.js:290:39
2018-02-08T14:48:50.8433486Z     at C:\npm\node_modules\newman\node_modules\postman-runtime\lib\backpack\index.js:56:34
2018-02-08T14:48:50.8433486Z     at C:\npm\node_modules\newman\node_modules\postman-runtime\lib\runner\index.js:96:20
2018-02-08T14:48:50.8433486Z     at extractRunnableItems (C:\npm\node_modules\newman\node_modules\postman-runtime\lib\runner\extract-runnable-items.js:135:35)
2018-02-08T14:48:50.8433486Z     at PostmanCollectionRunner.run (C:\npm\node_modules\newman\node_modules\postman-runtime\lib\runner\index.js:87:9)
2018-02-08T14:48:50.8433486Z     at C:\npm\node_modules\newman\lib\run\index.js:110:16
2018-02-08T14:48:50.8433486Z     at C:\npm\node_modules\newman\lib\run\options.js:246:9
2018-02-08T14:48:50.8433486Z     at C:\npm\node_modules\newman\node_modules\async\dist\async.js:473:16
2018-02-08T14:48:50.8433486Z     at C:\npm\node_modules\newman\node_modules\async\dist\async.js:3674:9 errno: -4068, code: 'EISDIR', syscall: 'read' }
2018-02-08T14:48:50.8433486Z newman warning: "html" reporter could not be loaded.
2018-02-08T14:49:20.7189833Z ##[error]C:\npm\newman.cmd failed with return code: 1
2018-02-08T14:49:20.7189833Z ##[section]Finishing: Newman - Postman
2018-02-08T14:49:20.7189833Z ##[section]Starting: Post Job Cleanup
2018-02-08T14:49:20.7189833Z Cleaning any cached credential from repository: <RepoName> (Git)
2018-02-08T14:49:20.7189833Z ##[command]git remote set-url origin <RepoURL>
2018-02-08T14:49:20.7502345Z ##[command]git remote set-url --push origin <RepoURL>
2018-02-08T14:49:20.7658612Z ##[section]Finishing: Post Job Cleanup
2018-02-08T14:49:20.7814858Z ##[section]Finishing: Run Postman Tests

[sebcaps]

Hmmm. Strange.
I guess last error is related to html-template path (permission or non existent folder). Unfortunately no idea for the two other.
Does command successful when no 'reporters' option set (ie only collection and environment ?)

[codewithtyler]

Yes it runs when I don't set the the reporters option.

[sebcaps]

Hmmm I must admit that I do not have any clues...
I compared mine command line and do not see difference(s) that may explain your pb. Nor the fact that none of the reporter is OK.... What about cli as reporter ?

Should you have better eyes than me I share my commend line for ref.
2018-02-14T02:07:21.6617417Z [command]C:\Windows\system32\cmd.exe /D /S /C "C:\npm\prefix\newman.cmd run --ssl-client-cert D:\a\1\s --ssl-client-key D:\a\1\s --insecure --reporter-html-template D:\a\1\s -r "cli,junit" -n 1 --globals D:\a\1\s\PostmanTests\globals.postman_globals.json -e D:\a\1\s\PostmanTests\APITest.postman_environment.json D:\a\1\s\PostmanTests\Shared.postman_collection.json"

[carlowahlstedt]

Only thing that jumps at me is the multiple inputs here for you @sebcaps

-r "cli,junit"

but @tylerbhughes only has a single for any of his three sample outputs

-r json

[codewithtyler]

I just ran it with multiple inputs, first trying cli, json and then cli, junit. In both cases one of the first logs I got said the repoter couldn't be loaded. Here's the message for junit:

newman warning: " junit" reporter could not be loaded.

The test runs now, I'm just not getting any test results even though I've got a Publish Test Results task. I assume that's because the reporter isn't getting loaded for whatever reason.

[sebcaps]

Can you try tout remove Space after ,. Here I guess this is the point (?)

[carlowahlstedt]

Agreed, @tylerbhughes if you'll try

cli,junit

then I think you'll be in business.

[carlowahlstedt]

@sebcaps I had considered making these checkboxes originally, but opted for the flexibility of the input. Any thoughts on changing it so all possible options are checkboxes?

[sebcaps]

@carlowahlstedt sounds good too me :-)
Seems input text is too much flexible !

[sebcaps]

After a while... How to consider migration from text to chekbox for exisiting installation ?
New major version ?

[carlowahlstedt]

I say we just add that v3 is currently in "beta" until we get things ironed out, such at this type of change.

[codewithtyler]

What happened to version 2.0? In the market place it says the current version is
1.0.20.

[carlowahlstedt]

Well...the way they version is kinda odd. So there's the task (Newman) that has a version and then there's an extension package that has a version. You can put multiple tasks into an extension package. In this case, it's a 1 to 1 relation.

[carlowahlstedt]

@tylerbhughes Did we get this cleared up?

[codewithtyler]

Good question, I'm on 3 projects at work and I've never gotten back to this. I'll see if I can find some time to try it out.

[codewithtyler]

Finally got a chance to work on this issue this morning. Here's the summary from the build.

	executed	failed
iterations	1	0
requests	42	0
test-scripts	33	0
prerequest-scripts	0	0
assertions	74	29
total run duration: 25.1s
total data received: 265.89KB (approx)
average response time: 1266ms

I'm still not getting the results in the test tab though. Does anyone know what newman names the results file? My guess is that the default value for the Publish Test Results task doesn't match the file name that newman creates. It's currently looking for **\TEST-*.xml

[codewithtyler]

Nevermind I figured it out. So a couple things to note: If you're trying to get the Postman results to show up in the tests tab like I've been doing you should do the following:

1. On the task for the Postman tests mark the checkbox for Continue on Error.
2. Add a Publish Test Results task and have it search for the following test file **\newman-*.xml.

Do those 2 things and you're results should start showing up in the tests tab.

[sebcaps]

Good to read it works...
Maybe ReadMe still require update... Definitely value of Continue on Error impact !

[apjden585]

Hi..

Is it possible to export the results to PDF?

[carlowahlstedt]

A quick review for you @apjden585 and it looks like the docs don't mention anything about it.

[carlowahlstedt]

@sebcaps I've added a section about the failed test bit in the readme for future reference.