Add backoff mechanism to recover from APM Server transport failures

[jlvoiseux]

Motivation / Summary

This PR aims to implement a backoff mechanism to recover from APM Server transport failures, due to the APM server being unavailable. To meet this goal, the mechanism described in issue #131 is implemented.

Changes

In apm_server.go, a simple finiste state machine is implemented. The states are as follows :

• nominal : in this state, the extension is able to send APM data without any issue.
• failure : we enter this state upon detection of a transport error. In this state, no data is sent to the APM server (the data is queued back into the dedicated channel).
• waitingForNextAttempt : upon detection of a transport error, a timer (defined according to the APM Transport spec) is started. When it ends, this state is used to signal to the extension that it can try again to send data to the APM server.

Pending questions

• When should we stop trying to reconnect to the APM server ? How can we relate this mechanism to the ELASTIC_APM_DATA_FORWARDER_TIMEOUT_SECONDS config option.

[apmmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-03-22T12:51:00.525+0000

• Duration: 7 min 2 sec

Test stats 🧪

Test	Results
Failed	0
Passed	240
Skipped	6
Total	246

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[estolfo]

Overall this looks good and I'm approving but I have just a few minor comments.

[estolfo]

Just one minor request