Use retry logic for HTTP communication with IMDS #107

dongsupark · 2024-07-18T14:31:57Z

Use retry logic and max duration for HTTP communication with IMDS, creating a generic wrapper retry_loop.
Doing that, it is possible for azure-init to retry to send HTTP requests, when requests failed for some reason.

For now we define a fixed const for timeout, max retries, for simplicity.

Make imds::query() capable of receiving input URL, so that both main binary and unit tests can send requests to different endpoints.

Fixes #63

Testing done

TBD

anhvoms · 2024-07-19T11:50:53Z

Do we know how much this increases the final build size of azure-init?

src/main.rs

anhvoms

I'm not sure how much we gain from using azure sdk vs just implementing our own retry policy here. Can you verify the final code size?

src/main.rs

dongsupark · 2024-07-19T12:37:23Z

I'm not sure how much we gain from using azure sdk vs just implementing our own retry policy here. Can you verify the final code size?

debug:

without Azure SDK

69084 -rwxrwxr-x 2 dpark dpark 70734488 Jul 19 14:17 target/debug/azure-init*

with Azure SDK

114888 -rwxrwxr-x 2 dpark dpark 117639624 Jul 19 13:48 target/debug/azure-init*

+66%

release:

without Azure SDK

6904 -rwxrwxr-x 2 dpark dpark 7069624 Jul 19 14:19 target/release/azure-init*

with Azure SDK

10356 -rwxrwxr-x 2 dpark dpark 10603568 Jul 19 14:15 target/release/azure-init*

+49%

Tried also turning off default features of Azure SDK crates, but not much difference.

jeremycline

I'm on the fence about the SDK - the lack of error codes to retry seems like it might be a deal-breaker. Since we want a fixed retry policy without jitter and so on, the implementation could be a loop with a constant sleep in it, wrapped in https://docs.rs/tokio/latest/tokio/time/fn.timeout.html.

src/main.rs

dongsupark · 2024-07-25T12:42:20Z

I'm on the fence about the SDK - the lack of error codes to retry seems like it might be a deal-breaker. Since we want a fixed retry policy without jitter and so on, the implementation could be a loop with a constant sleep in it, wrapped in https://docs.rs/tokio/latest/tokio/time/fn.timeout.html.

Ok, let me rewrite this PR based on https://docs.rs/tokio/latest/tokio/time/fn.timeout.html or any simpler ones.

src/main.rs

libazureinit/src/util.rs

src/main.rs

jeremycline

A few very minor notes, but it looks good to me. You should also be able to write a unit test for the timeout case, I think, with https://docs.rs/tokio/latest/tokio/time/fn.advance.html although I've not personally done this before so I'm just guessing based on the documentation.

libazureinit/src/error.rs

libazureinit/src/imds.rs

src/main.rs

dongsupark · 2024-08-08T14:07:01Z

A few very minor notes, but it looks good to me. You should also be able to write a unit test for the timeout case, I think, with https://docs.rs/tokio/latest/tokio/time/fn.advance.html although I've not personally done this before so I'm just guessing based on the documentation.

Added unit tests using tokio::time::advance etc.

jeremycline

Looks great, thanks! I had two little optional notes, but this is good to go from my perspective.

libazureinit/src/imds.rs

libazureinit/Cargo.toml

vmarcella

For the most part this LGTM, but I do want @anhvoms thoughts on exponentially increasing the retry interval when making repeated calls to IMDS before approving these changes.

libazureinit/src/imds.rs

vmarcella · 2024-08-12T20:13:16Z

libazureinit/src/imds.rs

-        let imds_body = response.text().await?;
-        let metadata: InstanceMetadata = serde_json::from_str(&imds_body)?;
+                if statuscode.is_success() && statuscode == StatusCode::OK {
+                    return response.error_for_status();


What would error_for_status() return here given that the status code is a success?

In that case, it would just return the inner response of type Response itself, where the outer response would be Ok.

libazureinit/src/imds.rs

src/main.rs

Use retry logic and max duration for HTTP communication with IMDS, creating a retry loop over getting response from IMDS. Doing that, it is possible for azure-init to retry to send HTTP requests, when requests failed for some reason. For now we define a fixed const for timeout, max retries for simplicity. Make imds::query() capable of receiving input URL, so that both main binary and unit tests can send requests to different endpoints.

dongsupark force-pushed the imds-http-retry branch 3 times, most recently from 10f8dae to 7b58355 Compare July 19, 2024 11:48

anhvoms reviewed Jul 19, 2024

View reviewed changes

src/main.rs Outdated Show resolved Hide resolved

anhvoms reviewed Jul 19, 2024

View reviewed changes

src/main.rs Outdated Show resolved Hide resolved

anhvoms requested changes Jul 19, 2024

View reviewed changes

src/main.rs Outdated Show resolved Hide resolved

dongsupark force-pushed the imds-http-retry branch from 7b58355 to b04a3c2 Compare July 23, 2024 13:25

jeremycline reviewed Jul 24, 2024

View reviewed changes

src/main.rs Outdated Show resolved Hide resolved

src/main.rs Outdated Show resolved Hide resolved

src/main.rs Outdated Show resolved Hide resolved

dongsupark changed the title ~~Use Azure SDK for HTTP communication with IMDS~~ Use retry logic for HTTP communication with IMDS Jul 25, 2024

dongsupark force-pushed the imds-http-retry branch from b04a3c2 to c4d8a9a Compare July 25, 2024 15:17

jeremycline requested changes Jul 29, 2024

View reviewed changes

src/main.rs Outdated Show resolved Hide resolved

libazureinit/src/util.rs Outdated Show resolved Hide resolved

src/main.rs Outdated Show resolved Hide resolved

dongsupark force-pushed the imds-http-retry branch from c4d8a9a to a4559b5 Compare July 30, 2024 14:45

jeremycline reviewed Jul 30, 2024

View reviewed changes

libazureinit/src/error.rs Outdated Show resolved Hide resolved

libazureinit/src/imds.rs Outdated Show resolved Hide resolved

libazureinit/src/imds.rs Outdated Show resolved Hide resolved

src/main.rs Outdated Show resolved Hide resolved

dongsupark force-pushed the imds-http-retry branch 2 times, most recently from 0cf0ba6 to 86085a1 Compare August 8, 2024 14:05

dongsupark marked this pull request as ready for review August 8, 2024 14:05

dongsupark force-pushed the imds-http-retry branch from 86085a1 to 7a5e641 Compare August 8, 2024 14:08

jeremycline approved these changes Aug 12, 2024

View reviewed changes

libazureinit/src/imds.rs Outdated Show resolved Hide resolved

libazureinit/Cargo.toml Outdated Show resolved Hide resolved

vmarcella requested changes Aug 12, 2024

View reviewed changes

anhvoms approved these changes Aug 12, 2024

View reviewed changes

vmarcella approved these changes Aug 12, 2024

View reviewed changes

dongsupark force-pushed the imds-http-retry branch from 7a5e641 to c6f3627 Compare August 20, 2024 15:28

jeremycline approved these changes Aug 20, 2024

View reviewed changes

dongsupark merged commit 1f64f1c into Azure:main Aug 21, 2024
5 checks passed

dongsupark deleted the imds-http-retry branch August 21, 2024 08:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use retry logic for HTTP communication with IMDS #107

Use retry logic for HTTP communication with IMDS #107

dongsupark commented Jul 18, 2024 •

edited

Loading

anhvoms commented Jul 19, 2024

anhvoms left a comment

dongsupark commented Jul 19, 2024

jeremycline left a comment

dongsupark commented Jul 25, 2024

jeremycline left a comment

dongsupark commented Aug 8, 2024

jeremycline left a comment

vmarcella left a comment

vmarcella Aug 12, 2024

dongsupark Aug 20, 2024

Use retry logic for HTTP communication with IMDS #107

Use retry logic for HTTP communication with IMDS #107

Conversation

dongsupark commented Jul 18, 2024 • edited Loading

Testing done

anhvoms commented Jul 19, 2024

anhvoms left a comment

Choose a reason for hiding this comment

dongsupark commented Jul 19, 2024

jeremycline left a comment

Choose a reason for hiding this comment

dongsupark commented Jul 25, 2024

jeremycline left a comment

Choose a reason for hiding this comment

dongsupark commented Aug 8, 2024

jeremycline left a comment

Choose a reason for hiding this comment

vmarcella left a comment

Choose a reason for hiding this comment

vmarcella Aug 12, 2024

Choose a reason for hiding this comment

dongsupark Aug 20, 2024

Choose a reason for hiding this comment

dongsupark commented Jul 18, 2024 •

edited

Loading