Outlook API Throttling documentation

[rbarten]

We often receive an error 429 when using the Microsoft Graph or Outlook Mail REST API for particular actions on a mailbox. Error 429 seems to be a Throtting issue, but we cannot find any information about the issue we experience, because it's doesn't seems to be related to the 'Rate-Limit' throttling rule documented here: https://blogs.msdn.microsoft.com/exchangedev/2017/04/07/throttling-coming-to-outlook-api-and-microsoft-graph/

We are able to reproduce our issue with the following step below: (we used direct Outlook Mail API to reproduce it)

• Get number of items in the Inbox
• Get subfolders of the Inbox
• Read first 10 e-mails sorted by ‘received date’
• For each email, move the e-mail to the ‘processedbox’ (which is a subfolder of the Inbox).

At exactly 500 ‘move’ actions we see the following behavior of the API:

• After 500 move actions:
  o Error 500:
  code=ErrorMoveCopyFailed
  message=The move or copy operation failed.
• The next ‘move’-action
  o Error 429
  code=ErrorTooManyObjectsOpened
  message=Too many concurrent connections opened., The process failed to get the correct properties.
• Important to mention here is that the other 3 request (Get number of items, get subfolders and read first 10 e-mails) are still working after this and return the correct data. Only on the move action we receive an error 429.

We tried several mailboxes (shared and user mailboxes), same issue.

It’s not the ‘Rate Limit’-throttling, because the headers returns the information below and there are still remaining requests, when the error 500/429 occurs:
Rate-Limit-Limit: 10000
Rate-Limit-Remaining: 9502
Rate-Limit-Reset: 2018-01-12T15:30:26.817Z

We currently cannot find any other Throttling documentation related to the 500 move actions.

We sometimes have to wait for more than 30 minutes before the 429 error disappears.

@jasonjoh
Is this a bug in the API? Or a throttling rule/issue?
How can we solve this issue?
If it’s a throttling rule, could you provide us additional documentation on these rules?

Thanks!

[jasonjoh]

I don't know the answer off the top of my head, I'll have to take a look at your repro and see what I can find out.

[jasonjoh]

Update: I was able to reproduce this using Graph, though for me I get the first 500 at the 498th move, and other requests (list messages, list subfolders) also fail with 429. I'll send the traces to engineering and see what I can find out.

[rbarten]

Great! We noticed the same when using the Graph. All requests are failing then.
When we use the Outlook Mail API directly only the Move action fails with 429, the other requests (list messages and subfolders) are working fine.

[jasonjoh]

I've logged a bug on this. I'll update as I get more information.

[faboo]

Hi, I'm experiencing a similar issue - many moves in rapid succession followed by ErrorTooManyObjectsOpened errors that contradict the Rate-Limit headers - has there been any change in its status?

[jasonjoh]

@faboo no, I'm still waiting for the engineering team to finish investigating. I will definitely update here as soon as I have more information to share.

[davster]

While confusing, TooManyObjectsOpened is not a REST-level throttling error, it is a store level complaint which is why it doesn't seem to be in agreement with the backoff header stuff. Store limits the number of concurrent MAPI session that it will allow against a given database as well as the number of open objects that a given session can have at any given time. I don't remember the numbers off the top of my head, but it is likely you are hitting the latter. My guess is that in batch operations, these handles are held open concurrently. I would suggest trying a smaller batch size and see if that helps (and not going in parallel either as that would defeat the purpose).

[Danthar]

We are running into the exact same issue.

[lauriemoore]

I am having this problem too. Is there a timeframe on when it can be fixed? Are there any work arounds? It was suggested to me to try to move more than one message in a single request by having a comma separated list of the message ids in order to cut back on the number of requests being made. But, I haven't tried that yet. My current call looks like: Message movedMsg = await graphClient.Users[inbox].Messages[message.Id].Move(needsAttentionFolder).Request().PostAsync(); Using the API like this is supposed to clean up the connections and session for me. Otherwise, what calls do I make to keep the number of open objects cleaned up? Thank you!

[davster]

The issue here isn't the number of calls, it is the number of concurrently opened objects in the store. Batching is good for other reasons, but will actually just exasperate this problem here. How many messages are you trying to move in a single request? In theory at least, assuming the messages are moving serially on the server, you shouldn't have more than one open handle at a time. Ill take a look at the code to see if that is not the case. What I would try to do is as my above post suggests, try a smaller batch size and reduce parallelization and see if that helps in the meantime.

[lauriemoore]

I'm just moving one at at time. I get 20 messages at time and loop through them individually. For each one, I download the attachments if any and get the Transport Header because I need some information from it. And then, if my processing is successful, I move the message to a folder called Processed. Or, if not everything worked out, for example, we will only take certain attachment types... the email is moved to the Needs Attention folder. Then, the code does it for the next email. It's pretty simple. I am not sure I'm using parallelization and I think my batch size is pretty small.

[davster]

Yeah, that shouldn't be happening. Out of curiosity, do you log your response headers? There are a few response headers which would be helpful: request-Id, the UTC time of the request and the X-BEServer name.

[Danthar]

So we are doing practically the same thing as @lauriemoore is doing. And right now we are consistently running into this issue. Its annoying because the retry-after response headers are not communicated back by the Graph SDK.

What happens is that a ServiceException is thrown. These are the relevant response headers communicated back.

request-id: 50b6fb6d-2226-4c51-9c7f-117ea4a0931b
x-ms-ags-diagnostic: {"ServerInfo":{"DataCenter":"West Europe","Slice":"SliceA","Ring":"4","ScaleUnit":"000","Host":"AGSFE_IN_27","ADSiteName":"AMS"}}
Date: Thu, 08 Mar 2018 10:48:59 GMT

[Danthar]

I reproduced the issue using a plain old HttpClient. And verified that the retry-after headers are not present, even though we are getting a 429 error.
Example url (if that would help): https://graph.microsoft.com/v1.0/users/[REDACTED]/mailFolders/Inbox/messages?$expand=Attachments&top=10
Ping: @davster

[rbarten]

I think you should use the Outlook Rest API directly instead of the Graph to receive those headers.
We also are receiving the error more often, But I haven’t found the time to retrieve the headers. We are also using Graph, But Outlook API directly has the same issue.

[rbarten]

@davster Below two samples of failing requests. First we get a error 500, after that 429 on each request:
I could provide the complete Fiddler trace if you want.

Error 500
Request:

POST https://outlook.office.com/api/v2.0/users/[mailbox]/messages/AAMkAGFlNWI3NzRlLWE2ZTAtNDc4MC04ZjQ2LTExOGFjM2RmODgzZgBGAAAAAADpRQjar8OpQ5QMu1_bHJVkBwCGA1ST-vJpQaEEt_WJYDxbAAAAAAEMAACGA1ST-vJpQaEEt_WJYDxbAAF2fI0iAAA=/move HTTP/1.1
Authorization: Bearer ....
Accept: application/json, application/xml, text/json, text/x-json, text/javascript, text/xml
User-Agent: RestSharp/106.2.1.0
Content-Type: application/json; charset=utf-8
Host: outlook.office.com
Content-Length: 140
Accept-Encoding: gzip, deflate

{"DestinationId":"AAMkAGFlNWI3NzRlLWE2ZTAtNDc4MC04ZjQ2LTExOGFjM2RmODgzZgAuAAAAAADpRQjar8OpQ5QMu1_bHJVkAQCGA1ST-vJpQaEEt_WJYDxbAAGYwSmRAAA="}

Response

HTTP/1.1 500 Internal Server Error
Cache-Control: private
Content-Type: application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8
Server: Microsoft-IIS/10.0
request-id: b150dc27-d6d2-4a8d-a2ac-4b476dd0310b
X-CalculatedFETarget: AM3PR05CU003.internal.outlook.com
X-BackEndHttpStatus: 500
Set-Cookie: exchangecookie=ed26572407b948cb878641c9019c155b; expires=Sat, 09-Mar-2019 14:32:47 GMT; path=/; HttpOnly
X-FEProxyInfo: AM3PR05CA0102.EURPRD05.PROD.OUTLOOK.COM
X-CalculatedBETarget: AM3PR07MB0661.eurprd07.prod.outlook.com
X-BackEndHttpStatus: 500
x-ms-appId: 940b9fc4-037c-47d9-b647-cf70495cadc7
Rate-Limit-Limit: 10000
Rate-Limit-Remaining: 9295
Rate-Limit-Reset: 2018-03-09T14:34:00.279Z
OData-Version: 4.0
X-AspNet-Version: 4.0.30319
X-DiagInfo: AM3PR07MB0661
X-BEServer: AM3PR07MB0661
X-FEServer: AM3PR05CA0102
X-Powered-By: ASP.NET
X-FEServer: VI1PR0801CA0085
Date: Fri, 09 Mar 2018 14:32:47 GMT
Content-Length: 87

{"error":{"code":"ErrorMoveCopyFailed","message":"The move or copy operation failed."}}

Error 429
Request:

POST https://outlook.office.com/api/v2.0/users/[mailbox]/messages/AAMkAGFlNWI3NzRlLWE2ZTAtNDc4MC04ZjQ2LTExOGFjM2RmODgzZgBGAAAAAADpRQjar8OpQ5QMu1_bHJVkBwCGA1ST-vJpQaEEt_WJYDxbAAAAAAEMAACGA1ST-vJpQaEEt_WJYDxbAAF2fI0kAAA=/move HTTP/1.1
Authorization: Bearer ...
Accept: application/json, application/xml, text/json, text/x-json, text/javascript, text/xml
User-Agent: RestSharp/106.2.1.0
Content-Type: application/json; charset=utf-8
Host: outlook.office.com
Content-Length: 140
Accept-Encoding: gzip, deflate

{"DestinationId":"AAMkAGFlNWI3NzRlLWE2ZTAtNDc4MC04ZjQ2LTExOGFjM2RmODgzZgAuAAAAAADpRQjar8OpQ5QMu1_bHJVkAQCGA1ST-vJpQaEEt_WJYDxbAAGYwSmRAAA="}

Response:

HTTP/1.1 429
Cache-Control: private
Content-Type: application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8
Server: Microsoft-IIS/10.0
request-id: 3ebc5450-ed36-4356-95a2-93374fab4709
X-CalculatedFETarget: AM3PR05CU003.internal.outlook.com
X-BackEndHttpStatus: 429
Set-Cookie: exchangecookie=10886bfe091a4530be1227d6629300db; expires=Sat, 09-Mar-2019 14:32:49 GMT; path=/; HttpOnly
X-FEProxyInfo: AM3PR05CA0090.EURPRD05.PROD.OUTLOOK.COM
X-CalculatedBETarget: AM3PR07MB0661.eurprd07.prod.outlook.com
X-BackEndHttpStatus: 429
x-ms-appId: 940b9fc4-037c-47d9-b647-cf70495cadc7
Rate-Limit-Limit: 10000
Rate-Limit-Remaining: 9293
Rate-Limit-Reset: 2018-03-09T14:34:00.279Z
OData-Version: 4.0
X-AspNet-Version: 4.0.30319
X-DiagInfo: AM3PR07MB0661
X-BEServer: AM3PR07MB0661
X-FEServer: AM3PR05CA0090
X-Powered-By: ASP.NET
X-FEServer: VI1PR0801CA0085
Date: Fri, 09 Mar 2018 14:32:49 GMT
Content-Length: 149

{"error":{"code":"ErrorTooManyObjectsOpened","message":"Too many concurrent connections opened., The process failed to get the correct properties."}}

[lauriemoore]

Does anyone have an update on this? We run this in production and our quarter end is quickly approaching and our email volume will go up. Today, we started getting in more emails at 8am... and by 3pm we had hit the 500 limit. In a few days, we will get 300 emails in 30 mins...and this will become a huge problem for us. We have our scheduling program wait for 30 mins before running the application again... and now even doing this is not resetting it. (Not even sure what "it" is.) We converted this process from Lotus Notes in Feb to Outlook365 and it is really hurting us.

Has anyone found a work around? If waiting 30 mins isn't working anymore... I wonder how long we have to wait? We will get so backlogged. Is there any word on when it will get fixed? 500 emails over 7 hours should not cause us to be throttled.

Thank you so much! Anxiously waiting for some good news. -Laurie

[davster]

Known issue that was recently fixed (after I responded above). The issue wasn't with traditional throttling. Basically there was an object leak that was happening for Move commands which caused us to exhaust the available handles and store started yelling at us. These leaked handles will affect any other call for that user.

Anyways, you should start seeing relief.

[lauriemoore]

I wondered if something had changed. We almost feel like the problem is worse now. It is inconsistent and hard to remedy.

I think I can get about 700 mail moves before I get the error. But, then, waiting 30 minutes is not working and I don't know how long it will take before the problem resets because the retry-after is not populated. It seems like waiting 30 minutes is a really long time anyway.

What type of volume should we be able to get through?

[jasonjoh]

@rbarten @lauriemoore This particular issue should be fixed. I just ran my repro app through 1200 messages with no error.

[gregory]

We are trying to import the entire inbox by using the batch endpoint of the graph service.
Out of 20 requests batched together, where each retrieve 100 messages, the response consistently
return with 4 with ErrorTooManyObjectsOpened.
Here below a sample of the response coming back:

What am i doing wrong @Danthar @jasonjoh

  "id": "16",
  "status": 429,
  "headers": {
    "Cache-Control": "private"
  },
  "body": {
    "error": {
      "code": "ErrorTooManyObjectsOpened",
      "message": "Too many concurrent connections opened., Cannot open mailbox.",
      "innerError": {
        "request-id": "990bfd1e-6a43-41b9-985d-42f42b014c23",
        "date": "2018-06-20T17:00:26"
      }
    }
  }
}```

[tit1]

I've been performing a batch action, creating 10 email messages and receive 2 back with the same error as @gregory . Its interesting to note that there is no retry after header.

[Prasunakakani]

Can any one give any idea or suggestion to resolve this, even i am facing the same error with just couple of requests.

[Herz3h]

I have the same issue on the Microsoft Graph Calendar API (adding new event). It fails really early (4th batch, requests n°62) and randomly, currently sending 350 requests split into batches of 20 requests. The response status code is 429 and the response is like this :

array:4 [▼
  "id" => "16"
  "status" => 429
  "headers" => array:1 [▼
    "Cache-Control" => "private"
  ]
  "body" => array:1 [▼
    "error" => array:3 [▼
      "code" => "ErrorTooManyObjectsOpened"
      "message" => "Cannot open mailbox."
      "innerError" => array:2 [▼
        "request-id" => "6e5dcce0-b3c2-4ab4-aeb2-79519f59a6b4"
        "date" => "2018-09-18T08:44:03"
      ]
    ]
  ]
]

[jtiltx]

@Herz3h
I also am having the same issue when adding events. (Batch inserting and deleting).

   [id] => 4286dc9a505d4925e178b1d440fc76b1
   [status] => 429
   [headers] => stdClass Object
       (
           [Cache-Control] => private
       )

   [body] => stdClass Object
       (
           [error] => stdClass Object
               (
                   [code] => ErrorTooManyObjectsOpened
                   [message] => Cannot open mailbox.
                   [innerError] => stdClass Object
                       (
                           [request-id] => c0066de7-aac8-4e52-8eaf-c0c1875261ba
                           [date] => 2018-09-21T16:26:15
                       )

               )

       )

[Herz3h]

@jtiltx I fixed it by sending 16 elements instead of 20 per batch ;)

[jtiltx]

@Herz3h thanks for the pro tip, Ill give it a go. :)