Skip to content

SDK doesn't appear to use Retry-After on 429s #978

Closed
@gavinbarron

Description

@gavinbarron
Member

Bug Report

Prerequisites

  • Can you reproduce the problem?
    Are you running the latest version?
    Are you reporting to the correct repository?
    Did you perform a cursory search?

Description

During testing of a tool to simulate 429s on the network layer (i.e. outside of the JavaScript library) it was observed that a when a Retry-After or 5s was returned to the client application the SDK issued a retry request before the 5s window had elapsed

Console Errors: N/A

Screenshots:

Steps to Reproduce

  1. Use the ChaosProxy sample using the sdk and with the proxy running with 99% failure rate and only 429s as the allowed errors
  2. see that the a retry is sent before the 5s wait period is finished

Expected behavior: [What you expected to happen]
SDK should not be retrying until after the number of seconds provided in the Retry-After header

Actual behavior: [What actually happened]
SDK retries the request at 3s and eventually succeeds due to the exponential backoff behavior

Additional Context

Add any other context about the problem here..
Chaos Proxy is here: https://github.com/microsoftgraph/msgraph-chaos-proxy (Microsoft private repo at present)

Usage Information

Request ID - N/A

SDK Version - script link to https://cdn.jsdelivr.net/npm/@microsoft/microsoft-graph-client/lib/graph-js-sdk.js so I assume latest

  • Browser (Check, if using Browser version of SDK)

Browser Name - Edge

Version - 105

Activity

self-assigned this
on Oct 26, 2022
waldekmastykarz

waldekmastykarz commented on Nov 14, 2022

@waldekmastykarz

This seems to be caused by the fact that the retry-after header is not listed among the safe headers in the access-control-expose-headers response header coming from Graph.

Because calling Graph from client-side apps is a CORS request, clients are only allowed to access a predefined set of safe headers, plus additional headers defined by the server as safe using the access-control-expose-headers header. Right now, retry-after is not listed among these headers, which is why the fetch API always returns its value as null.

To fix this issue, we'd need to add the retry-after header to the access-control-expose-headers header on Graph.

sebastienlevert

sebastienlevert commented on Nov 29, 2022

@sebastienlevert
Contributor

Quick update on this issue. We are working with the AGS team to add this header to the access-control-expose-headers. When available worldwide, we'll update this thread.

altano

altano commented on Oct 23, 2023

@altano

@sebastienlevert I'm not seeing this header in the response to graph API requests. I should expect to see it, right? (I assume you closing the topic means it was exposed at some point)

I do see retry-after listed in access-control-expose-headers but the header still isn't in the response, as you can see:
image

sebastienlevert

sebastienlevert commented on Oct 23, 2023

@sebastienlevert
Contributor

Some APIs might not send a "Retry-After" and the SDK will automatically backoff exponentially.

altano

altano commented on Oct 24, 2023

@altano

@sebastienlevert are you sure about the exponential backoff? Here's what I'm seeing:

image

Note the timestamps. Retries seem to be happening every few seconds without any backoff.

sebastienlevert

sebastienlevert commented on Oct 24, 2023

@sebastienlevert
Contributor

Based on this, it should https://github.com/microsoftgraph/msgraph-sdk-javascript/blob/dev/src/middleware/RetryHandler.ts#L136. What is the version of the SDK you are using?

altano

altano commented on Oct 26, 2023

@altano

I'm on @microsoft/microsoft-graph-client v3.0.7 (latest).

I realized what is going on: retryAttempts is capping out at 3 because of the default retry limit, and then the request fails. Then my code moves on to the next page and it does the same thing, with retryAttempts going from 0 - 3.

I am trying to fix this in my code by having a much higher maxRetries but I can't figure out how. Do I really have to create a middleware chain that replaces the default RetryHandler with a custom one, and then use that in Client.initWithMiddleware, or is there an easier way to control this?

FWIW I think it would make sense to have a MUCH higher default maxRetries value than 3 for 429 responses that don't include the retry-after header, as the first 3 retries are almost certainly going to also fail.

7 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    Participants

    @altano@gavinbarron@sebastienlevert@waldekmastykarz@nikithauc

    Issue actions

      SDK doesn't appear to use Retry-After on 429s · Issue #978 · microsoftgraph/msgraph-sdk-javascript