[AZI-529] improve log forwarder retry behavior

[parsons90]

What does this PR do?

Retries sending logs if the HTTP request failed. It will retry up to 4 times, with a 1 second timeout between retries, doubling each time. It will not retry http 4xx responses.

Motivation

The network connection can be flaky and it is worth retrying a failed request.

Testing Guidelines

Ran unit tests and tested in the Azure portal.

To test request errors, I constructed a bad http request with invalid options, and saw it attempt to retry the request. You can see the retry backoff in the timestamps.

Retrying based on status code. It doesn't retry on a 404, but does retry with a 500.

There was also a random timeout that got retried:

Additional Notes

Is 4 retries too much/not enough? How is a 1 sec timeout interval? Will doubling the timeout cause it to wait too long?

Without using external libraries, the only way to add a delay is with setTimeout, which requires a callback to be provided.

Types of changes

• Bug fix
• New feature
• Breaking change
• Misc (docs, refactoring, dependency upgrade, etc.)

Check all that apply

• This PR's description is comprehensive
• This PR contains breaking changes that are documented in the description
• This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
• This PR impacts documentation, and it has been updated (or a ticket has been logged)
• This PR's changes are covered by the automated tests
• This PR collects user input/sensitive content into Datadog
• This PR passes the integration tests (ask a Datadog member to run the tests)
• This PR passes the unit tests
• This PR passes the installation tests (ask a Datadog member to run the tests)

[gpalmz]

LGTM. Maybe worth getting another set of eyes since we don't work on this much