Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add exponential backoff to the retry algorithm of WebCmdlets #19637

Closed
wants to merge 8 commits into from
Closed

Add exponential backoff to the retry algorithm of WebCmdlets #19637

wants to merge 8 commits into from

Conversation

mkht
Copy link
Contributor

@mkht mkht commented May 10, 2023 •
edited

PR Summary

Add a new parameter -RetryMode to Invoke-WebRequest and Invoke-RestMethod to add the ability to retry with exponential backoff strategy and exponential backoff with jitter strategy.

The type of -RetryMode is [WebRequestRetryMode] enum and users can select one from three modes.

  • Fixed: Use fixed retry interval. The interval follows the value of -RetryIntervalSec. This is the default mode.

  • Exponential: Use exponential backoff strategy. The retry interval is expressed as RetryIntervalSec * (2 ^ retryCount).

  • ExponentialJitter: Use exponential backoff with jitter strategy. The retry interval is expressed as RetryIntervalSec * (2 ^ retryCount) * jitter. jitter is a random value betweem 0.0 and 1.0.

Note 1:
If we select Exponential or ExponentialJitter, the interval will never exceed 600 seconds. This prevents the interval from becoming excessively long. 600 seconds is used by curl.

Note 2:
This is not a breaking change, since the default mode is Fixed. It does not change the behavior of existing scripts.

PR Context

This PR resolves #19632

PR Checklist

@mkht mkht requested a review from PaulHigin as a code owner May 10, 2023 14:14
@ghost ghost assigned TravisEz13 May 10, 2023
@iSazonov iSazonov marked this pull request as draft May 10, 2023 17:08
@iSazonov
Copy link
Collaborator

Convert to Draft until #19632 will be approved by WG.

mkht and others added 3 commits May 11, 2023 09:35
@mkht @CarloToso
Update src/Microsoft.PowerShell.Commands.Utility/commands/utility/Web…
837c462 
…Cmdlet/Common/WebRequestPSCmdlet.Common.cs

Co-authored-by: CarloToso <105941898+CarloToso@users.noreply.github.com>
@mkht @CarloToso
Update src/Microsoft.PowerShell.Commands.Utility/commands/utility/Web…
ba5d55b 
…Cmdlet/Common/WebRequestPSCmdlet.Common.cs

Co-authored-by: CarloToso <105941898+CarloToso@users.noreply.github.com>
@mkht
Fix type conversion error
a4b1b49
@CarloToso
Copy link
Contributor

CarloToso commented May 11, 2023 •
edited

@mkht You could use MathF.ScaleB https://learn.microsoft.com/en-us/dotnet/api/system.mathf.scaleb?view=net-7.0

@CarloToso
Copy link
Contributor

We could think about using an enum for RetryMode and a switch expression for retryIntervalInSeconds

@mkht
Copy link
Contributor Author

mkht commented May 13, 2023

We could think about using an enum for RetryMode and a switch expression for retryIntervalInSeconds

I was thinking that too. But I couldn't decide which was better, so I implemented it as a string.
if it is to be an enum, would naming it something like WebCmdletRetryMode or WebRequestRetryMode be good?

@CarloToso
Copy link
Contributor

I think WebRequestRetryMode could work. If we find something better in the future changing the name will be easy

@pull-request-quantifier-deprecated
Copy link

This PR has 83 quantified lines of changes. In general, a change size of upto 200 lines is ideal for the best PR experience!

Quantification details 

Label      : Small
Size       : +76 -7
Percentile : 33.2%

Total files changed: 2

Change summary by file extension:
.cs : +22 -7
.ps1 : +54 -0

Change counts above are quantified counts, based on the PullRequestQuantifier customizations.

Why proper sizing of changes matters

Optimal pull request sizes drive a better predictable PR flow as they strike a
balance between between PR complexity and PR review overhead. PRs within the
optimal size (typical small, or medium sized PRs) mean:

  • Fast and predictable releases to production:
    • Optimal size changes are more likely to be reviewed faster with fewer
      iterations.
    • Similarity in low PR complexity drives similar review times.
  • Review quality is likely higher as complexity is lower:
    • Bugs are more likely to be detected.
    • Code inconsistencies are more likely to be detected.
  • Knowledge sharing is improved within the participants:
    • Small portions can be assimilated better.
  • Better engineering practices are exercised:
    • Solving big problems by dividing them in well contained, smaller problems.
    • Exercising separation of concerns within the code changes.

What can I do to optimize my changes

  • Use the PullRequestQuantifier to quantify your PR accurately
    • Create a context profile for your repo using the context generator
    • Exclude files that are not necessary to be reviewed or do not increase the review complexity. Example: Autogenerated code, docs, project IDE setting files, binaries, etc. Check out the Excluded section from your prquantifier.yaml context profile.
    • Understand your typical change complexity, drive towards the desired complexity by adjusting the label mapping in your prquantifier.yaml context profile.
    • Only use the labels that matter to you, see context specification to customize your prquantifier.yaml context profile.
  • Change your engineering behaviors
    • For PRs that fall outside of the desired spectrum, review the details and check if:
      • Your PR could be split in smaller, self-contained PRs instead
      • Your PR only solves one particular issue. (For example, don't refactor and code new features in the same PR).

How to interpret the change counts in git diff output

  • One line was added: +1 -0
  • One line was deleted: +0 -1
  • One line was modified: +1 -1 (git diff doesn't know about modified, it will
    interpret that line like one addition plus one deletion)
  • Change percentiles: Change characteristics (addition, deletion, modification)
    of this PR in relation to all other PRs within the repository.

Was this comment helpful? 👍  :ok_hand:  :thumbsdown: (Email)
Customize PullRequestQuantifier for this repository.

@ghost ghost added the Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept label May 18, 2023
@StevenBucher98 StevenBucher98 added the PowerShell-Docs needed The PR was reviewed and a PowerShell Docs update is needed label May 22, 2023
@ghost ghost added the Stale label Jun 6, 2023
@ghost
Copy link

ghost commented Jun 6, 2023

This pull request has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 15 days. It will be closed if no further activity occurs within 10 days of this comment.

@mkht
Copy link
Contributor Author

mkht commented Jun 7, 2023

This PR is marked as draft as it is still awaiting approval of the proposal by the WG.

@ghost ghost added Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept and removed Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept Stale labels Jun 7, 2023
@ghost ghost added the Stale label Jun 25, 2023
@ghost
Copy link

ghost commented Jun 25, 2023

This pull request has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 15 days. It will be closed if no further activity occurs within 10 days of this comment.

@ghost ghost closed this Jul 5, 2023
@CarloToso
Copy link
Contributor

@SteveL-MSFT could you reopen this PR and review it

This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@turbedi turbedi turbedi left review comments

@CarloToso CarloToso CarloToso left review comments

@PaulHigin PaulHigin Awaiting requested review from PaulHigin

Assignees

@TravisEz13 TravisEz13

Labels
PowerShell-Docs needed The PR was reviewed and a PowerShell Docs update is needed Small Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add exponential backoff to the retry algorithm of WebCmdlets
6 participants
@mkht @iSazonov @CarloToso @turbedi @TravisEz13 @StevenBucher98