Azure helper: Increase Azure Endpoint HTTP retries

[johnsonshi]

Proposed Commit Message

Azure helper: Increase Azure Endpoint HTTP retries

Increase Azure Endpoint HTTP retries to handle
occasional platform network blips.

Introduce a common method http_with_retries
in the azure.py helper, which will serve as
the common HTTP request handler for
all HTTP requests with the Azure endpoint.
This method has builtin retries and
reporting diagnostics logic.

Additional Context

Additional context above.

Test Steps

Ensure no regressions in pre-existing Azure endpoint interaction for happy path code

• Create an Azure VM from the latest Ubuntu 18.04 LTS image.
• Install this PR's cloud-init.
• Capture the VM into a custom image.
• Deploy a VM from this custom image. Deployment succeeds.
• No warnings/errors. Manually inspected cloud-init logs and ensured interaction with the Azure endpoint was seamless.

Ensure that long delay log message is logged, if there is a long delay (>5 sec) that eventually succeeds during interaction with the Azure endpoint.

• Same steps as above. Modify http_with_retries to fail during the first 15 attempts, and only succeed at the 16th attempt.
• Capture the VM into a custom image.
• Deploy a VM from this custom image. Deployment succeeds.
• No warnings/errors. Manually inspected cloud-init logs and ensured interaction with the Azure endpoint succeeded but with the new success diagnostic message present.

Ensure that error messages are logged upon failure despite max attempt retries

• Same as above, except modify http_with_retries so that it fails after max attempts.
• Inspect cloud-init logs to ensure that the new failure diagnostic message is present.

Checklist:

• My code follows the process laid out in the documentation
• I have updated or added any unit tests accordingly
• I have updated or added any documentation accordingly

[johnsonshi]

Pending internal discussions on what the error messages should be.

[mitechie]

@johnsonshi is this ready for review? Are we happy with the error messages and such at this point?

Thanks!

[johnsonshi]

@mitechie yes we've discussed internally and these will be the final error messages.

[blackboxsw]

Thank you so much @johnsonshi for this branch. Minor nits on review. If this is acceptable, I can apply them or you can then we will land it.

[johnsonshi]

Thanks you so much @blackboxsw! I've addressed the requested changes above. Let's wait for #594 to land before merging this.

#594 is also about to be merged. Just addressed OddBloke's final set of minor requested changes and NITs in that other PR, so it should be mergeable once he reviews it.

Given that #594 is more complicated than this PR, it will be easier to fix this PR if there are any merge conflicts.

Also as a final validation, I built a deb package from this and performed Azure deployment tests. Confirmed that everything works fine with live deployment tests.

[blackboxsw]

Thanks @johnsonshi this it good to land today.