New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: Run test in parallel #744

Closed
passsy opened this Issue Nov 9, 2016 · 13 comments

Comments

Projects
None yet
8 participants
@passsy

passsy commented Nov 9, 2016

Sadly the webservice I'm testing has some very slow calls. My test suite runs 30min+ when I wait for responses. When I'd be able to run test in parallel it would be much faster! Running folders in parallel would be enough.

@a85

This comment has been minimized.

Contributor

a85 commented Nov 10, 2016

You can do this with multiple process instances?

@czardoz

This comment has been minimized.

Collaborator

czardoz commented Nov 10, 2016

Yep, that would be one solution. Another solution is to use Newman as a library in a simple script. We have an example for parallel collection runs here: https://github.com/postmanlabs/newman/blob/896a7b61955ee59bc8bfcfa144e7c5aa3639194e/examples/parallel-collection-runs.js

All you need to change in this script is the options, where you would want to specify different options for each run (each containing a different folder name)

@passsy

This comment has been minimized.

passsy commented Nov 10, 2016

Is one of both solutions possible to publish combined results (lint, json)?

@czardoz

This comment has been minimized.

Collaborator

czardoz commented Nov 10, 2016

Yes, but both involve writing some aggregation logic. For the first approach, you can use a JSON reporter, and aggregate results from the reports when all the parallel runs are complete.
For the second approach, it's a little easier, since you can aggregate results as they are available (i.e, during the run)

@czardoz

This comment has been minimized.

Collaborator

czardoz commented Nov 10, 2016

@passsy To add to my comment above, you can use the result.run object to find additional information about failures, successes etc 😄

@simonmbcgdv

This comment has been minimized.

simonmbcgdv commented Dec 10, 2016

@czardoz i'm running this script with 10 parallelCollectionRuns on newman 3.3.0 on linux and notice the following behaviour:

  1. it's getting noticeably slower over the course of about 1h towards the tail end. I've run out of memory twice and increased the node old memory accordingly. I suspect it may be leaking memory?

  2. While running postman scripts using parallelCollectionRuns, we have the old {{ unsubstituted variable }} sent to the server during execution, for a small fraction of requests only. This never happens when we run the script single threaded from the CLI using newman directly, where everything is substituted correctly. It looks like a side effect of the parallelCollectionRun.

anybody else experience this? If you need specs / further details please let me know.

Cheers,

Simon

@shamasis

This comment has been minimized.

Member

shamasis commented Dec 10, 2016

Just to add my two cents - Newman has this behaviour of growing memory usage owing to the amount of details it currently needs to hold on in memory. Down the line, we need to possibly run Newman in a mode where it flushes out results to disk and not store in memory. Until then, parallel or no parallel, long running and repeated requests will have this issue.

@sumit16986

This comment has been minimized.

sumit16986 commented Jan 4, 2018

Please provide the parallel run feature to the newman as this is the most useful feature when some has the some huge amount of test cases........

@gas421

This comment has been minimized.

gas421 commented Mar 16, 2018

@shamasis please consider this feature into the roadmap of Newman. I would like to point out the parallel execution will be really useful in the context of be able to speedup tests and have faster feedbacks (NOT for load testing purpose).
Thanks!

@flash286

This comment has been minimized.

flash286 commented Apr 5, 2018

I'm voting for parallel run and it's gonna be fantastic useful.
Until newman implementing this, i did small tool written on golang around it which can help to run it in parallel mode and get aggregated stats. But it still very experimental.
https://github.com/flash286/postman-load-testing

@gas421

This comment has been minimized.

gas421 commented Apr 6, 2018

Hello, thanks @flash286 for your message even if I have a little concern: I think the overall idea and vision of Postman is not to be also a Load Test tool, for this you have many other different tools and option. I see Postman as an API functional validation tool.
My message was focused more on the integration with Postman Newman and Jenkins (or TFS...). My point was to have a possibility to run in parallel the API functional validation tests step (i.e. at the end of a build sequence) in order to speed up the execution of the whole build.

Nothing to do with load testing.

Thanks!

@shamasis

This comment has been minimized.

Member

shamasis commented Jun 5, 2018

This is being tracked in main postman-app-support tracker. This feature is needed, but not exactly in the way we are thinking of. We would need to think more on this. 🤔

@shamasis shamasis closed this Jun 5, 2018

@gas421

This comment has been minimized.

gas421 commented Jun 6, 2018

@shamasis can you post the link to this "main postman-app-support tracker"?
Is there a way to follow and get updates about this topic?
How are you going to proceed?
Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment