Retry 5xx failures

[janpieterz]

We've had a couple of builds fail today because the Node tool installer returned a 502 response.
Rerunning it fixed it. It could be interesting if it fails with a 5xx response to wait a second, try again, wait 3 seconds, try again, then fail.

Relevant lines (as far as I can see):
https://github.com/Microsoft/vsts-task-tool-lib/blob/master/tool.ts#L225

Let me know if you'd appreciate this and I can make a pull request.

Relevant logs:

2018-01-06T14:00:55.4185057Z Downloading: https://nodejs.org/dist/v8.9.4/node-v8.9.4-win-x64.7z
2018-01-06T14:00:55.4185442Z ##[debug]destination d:\a\_temp\c35d5f0d-b2b7-43ed-8309-0e1078420071
2018-01-06T14:00:55.4186351Z ##[debug]creating stream
2018-01-06T14:00:55.4208841Z ##[debug]downloading
2018-01-06T14:00:55.4437058Z ##[debug]task result: Failed
2018-01-06T14:00:55.4533722Z ##[error]Unexpected HTTP response: 502
2018-01-06T14:00:55.4534242Z ##[debug]Processed: ##vso[task.issue type=error;]Unexpected HTTP response: 502
2018-01-06T14:00:55.4552145Z ##[debug]Processed: ##vso[task.complete result=Failed;]Unexpected HTTP response: 502

[stephenmichaelf]

@janpieterz Thanks for reaching out with the suggestion! IMO we wouldn't want to do this for most of the 5xx error codes like internal server error, service unavailable, gateway timeout, etc. These are server errors and I am not sure, in the general case, that retrying will help. Most of the time I would guess that they are issues that need to be fixed on the server and that aren't resolved by waiting a little bit.

@bryanmacfarlane What do you think?

[bryanmacfarlane]

I don't think we should retry on 500

But, I wouldn't be opposed to retrying on these since they are temporary states:
502 Bad Gateway
The server was acting as a gateway or proxy and received an invalid response from the upstream server.
503 Service Unavailable
The server is currently unavailable (because it is overloaded or down for maintenance). Generally, this is a temporary state.
504 Gateway Timeout
The server was acting as a gateway or proxy and did not receive a timely response from the upstream server.

[stephenmichaelf]

@janpieterz Would you like to make this change and do a PR? :)

[janpieterz]

Yup, be happy to. Give me some time as I'm finishing up some things.