Collection runner stops to run if one request fails.

[RuniQa]

App Details:

Postman for Windows
Version 5.3.2
win32 10.0.15063 / x64

Issue Report:

Description
One of my services (name it service_X) is off from time to time. In that case there I will not get response from it.

However rest of my services will respond with success. Service_X is tested somewhere in the middle of the collection. Like in the example:

When I run Collection Runner to check all requests in the collection and service_X is off, I am getting error and any next steps are not executed.

Expected behaviour:
Collection Runner should run to the end of the collection and log error in execution of one of the tests.

I would be grateful for any help on that matter.

[kamalaknn]

@RuniQa we do stop the run if a request errors out. Unfortunately, I do not see an option to disable this yet.

I am marking this as a feature request. In the meantime you can run the same collection using our command line tool newman with the --bail option set to false.

[Citizen01]

I have the same issue feature request.
One would expect the Collection runner to execute all the requests in a collection no matter what and then check the results to see which one are OK and which one are KO.
I'll use newman in the meantime and follow that issue.

[tomershay]

+1

[efrenette11]

+1

[skubot]

Also want to +1, especially... can the report show 1 fail. At the moment it reports 0 fail, so it looks like all tests passed.

[fredericpaladin]

+1

[jerrod-williams]

attempting to use this in production on a web server to verify connectivity. we prefer not to run node on our windows servers, so newman isn't preferred.

[modosteve]

+1

[rakeshk1987]

+1

[fjab76]

+1

[dzottola]

+1

[pavankumar-dv]

+1

[modosteve]

+1

[karlspotcap]

+1

[KilgallonJ]

+1

[ConnorChristie]

+1

[felipeauol]

+1

[sergeychubuk]

+1

[edin-music-sn]

+1

[aarystan]

+1

[rmaschre]

+1

[suhasbobba]

+1

[thedoseoftea]

Would also appreciate this.

[rsoudani]

+1

[nseth]

+1 It's so weird to offer this feature incomplete like this. We would like to build our our tests ahead of time and run while we develop - catching more than one bug/issue at a time.

[gauravkhuraana]

Strange to know this feature is still not there. If i have 100 test, and test nbr 2 fails.. I cannot check what others 98 are doing..

For now this has helped from command line, but this feature should be brought to UI. Its very basic

 newman run projectName.postman_collection.json --bail false

--bail false.. does not stop the test when ran from command line as mentioned by one of the user above

[gauravkhuraana]

@RuniQa we do stop the run if a request errors out. Unfortunately, I do not see an option to disable this yet.

I am marking this as a feature request. In the meantime you can run the same collection using our command line tool newman with the --bail option set to false.

@RuniQa anytime line to get this feature ?

[fransf-wtax]

@RuniQa Would really appreciate having this. As it stands, the behaviour is a bit all over the place: a test assertion fails? Tests continue. A request returns an error code? Tests continue. A Javascript error occurs in a Test? Tests stop.
At least give us some controls to set what should be cause for a test run to be stopped and what should be ignored.
If this is already possible in newman, is it really that hard to add this to the UI test runner?

[xaviergm]

+1 - I would like the runner to continue when it does not get a response at all from the server, in the same way it goes on when a 404 or a 500 status code is returned from the server. Not getting a response from the server is, in a way, a server error but that the server wasn't able to manage, right?

Could not get response - Error: read ECONNRESET halts all processing.

I would like to have the --bail option as a checkbox on the runner next to Save Responses, Keep variable values, etc. If it exists in Newman it could exist as well in Postman, right?

Thanks!

[LTRichHouse]

+1.

[varunet]

+1

[hcdbh14]

+1

[mgarcia-accesso]

+1

[lucianamerege]

+1 It's been 6 years since this was brought up, still nothing?

[hcdbh14]

Over the years I found a solution for this.
The reason why runs stop is because of the JavaScript scripts that fail. (syntax, missing objects and such)

If you cover all the code within pm.test, then exceptions will always be handled and the runner won't stop.

[malvikach]

Hi all,

To confirm you are looking for Runner to continue to execute if there is an error in a Request. I have a rough mockup to share, let me know if this makes sense, we can add an option in Advanced Settings to give you more control over your runs :

You can indicate if you wish to stop the run on request error or continue to run by checking/unchecking this option. We will remember this setting in your session. By default, we will go with the existing behavior i.e. to stop the run when error occurs.

Let me know if you like to add some thoughts on this.

Regards,
Malvika

[xaviergm]

To confirm you are looking for Runner to continue to execute if there is an error in a Request. I have a rough mockup to share, let me know if this makes sense, we can add an option in Advanced Settings to give you more control over your runs :
You can indicate if you wish to stop the run on request error or continue to run by checking/unchecking this option. We will remember this setting in your session. By default, we will go with the existing behavior i.e. to stop the run when error occurs.

This looks really good to me. Thanks!

[LTRichHouse]

Hi all,

To confirm you are looking for Runner to continue to execute if there is an error in a Request. I have a rough mockup to share, let me know if this makes sense, we can add an option in Advanced Settings to give you more control over your runs :

You can indicate if you wish to stop the run on request error or continue to run by checking/unchecking this option. We will remember this setting in your session. By default, we will go with the existing behavior i.e. to stop the run when error occurs.

Let me know if you like to add some thoughts on this.

Regards, Malvika

That looks great - thanks Malvika. Will Newman also support continued running when an error occurs?

[malvikach]

@LTRichHouse newman already supports this. By default, a newman run will continue to happen and go to the next request if the current request is in error. If you wish for the run to stop when an error occurs you can use the option --bail.

Hope this helps.

[RuniQa]

@malvikach Oh boy, that looks great.

[lucianamerege]

Over the years I found a solution for this. The reason why runs stop is because of the JavaScript scripts that fail. (syntax, missing objects and such)

If you cover all the code within pm.test, then exceptions will always be handled and the runner won't stop.

Can you please provide an example on how you cover the error within pm.test? I tried doing that but since the error sometimes comes from the request I think it doesn't even reach the testing phase.

[lucianamerege]

Hi all,

To confirm you are looking for Runner to continue to execute if there is an error in a Request. I have a rough mockup to share, let me know if this makes sense, we can add an option in Advanced Settings to give you more control over your runs :

You can indicate if you wish to stop the run on request error or continue to run by checking/unchecking this option. We will remember this setting in your session. By default, we will go with the existing behavior i.e. to stop the run when error occurs.

Let me know if you like to add some thoughts on this.

Regards, Malvika

Yes, that's great! Do you have a prediction on how long it would take to implement this?

[malvikach]

@lucianamerege I will keep this thread updated with timelines as I have more to share.

Regarding the errors inside pm.test(), I checked how Postman CLI or Newman handles it currently, it seems even if the request errors out, the assertions are run and result in assertion failures. (To reproduce an error inside the pm.test() I tried to access an object of a response, I also tried to just return an exit code of 1)

If you have some other examples regarding this, feel free to share and I can take a look.

Overall, how would you like the system to behave in case there is an error inside a pm.test() - do you expect the run to stop or continue? Currently, the behavior we will build will be similar to Newman/ Postman CLI so feel free to give it a spin on the terminal and let us know.

Thanks

[LTRichHouse]

Both postman client and Newman should continue to run further requests and tests. Both should clearly indicate that a failure has occurred during the test run in the test results. Newman should return a non-zero failure status.

I can only remember seeing the problem when using Windows Postman client (not Newman) , but my memory is bad.

[LTRichHouse]

@LTRichHouse newman already supports this. By default, a newman run will continue to happen and go to the next request if the current request is in error. If you wish for the run to stop when an error occurs you can use the option --bail.

Hope this helps.

Thanks. I had a feeling it did and it's reassuring to have this confirmed. I mentioned Newman as I wanted this "continuing" behaviour in Newman to continue by default - sounds like it will be.

For Postman client it would be even better if the option to continue was a persisted user setting that I would always have turned on, without having to remember to tick it in each session. I would almost always want continuation.

[malvikach]

Hi All,

Glad to inform everyone that this improvement has been released in the latest Postman release (version 10.13). You can now customize the Collection Runner to continue to run if an error occurs.

The setting will be checked by default i.e. by default Runner will stop the execution in case of an error. If you wish to continue runner to execute, you can uncheck this setting and Postman will remember it throughout your session for all subsequent runs unless you change it.

Hope this helps your workflows. Once again, thanks a lot for sharing your feedback on this.

Please note: The Postman version update is being rolled out, you will get it in the next 1-2 days if you haven't already.

@LTRichHouse Please do give this a try and let us know your experience, we can discuss further regarding the option to always remember the setting.

Cheers,
Malvika

[torgerss]

This is great, but not available in scratchpad mode...