Microsoft sync - Mailbox concurrency issues

[kar-re]

Bug Description

When syncing a Microsoft account, a MailboxConcurrency throttle issue is reported from the Microsoft Graph API.
Ex.

 MessageImportException [Error]: Unknown error occurred while importing messages for message channel 72dc8f2c-156e-45e7-8dbf-cba320346b92 in workspace e7a4c51f-4415-4cb9-b4ea-7def87c05649: Microsoft Graph API unknown error: Error: Microsoft Graph API error: Application is over its MailboxConcurrency limit.

This leads to the total import being canceled, instead of retrying after a while.

Expected behavior

Syncing should continue or retry after a fixed time

Technical inputs

The wierd thing is that it succeeds for three requests but then fails:

worker-1  | [Nest] 34  - 03/20/2025, 3:40:44 PM     LOG [MessagingMessageListFetchJob] Fetching full message list for workspace e7a4c51f-4415-4cb9-b4ea-7def87c05649 and account 26efae74-5545-4d82-b0aa-05e3187c2976
.
.
.
worker-1  | [Nest] 34  - 03/20/2025, 3:42:00 PM     LOG [MessagingMessagesImportService] Messaging import for workspace e7a4c51f-4415-4cb9-b4ea-7def87c05649 and account 26efae74-5545-4d82-b0aa-05e3187c2976 starting...
db-1      | 2025-03-20 15:42:03.145 UTC [24] LOG {ticks: 0, maint: 0, retry: 0}
db-1      | 2025-03-20 15:42:05,926 INFO: no action. I am (ccf62911b2bd), the leader with the lock
worker-1  | [Nest] 34  - 03/20/2025, 3:42:12 PM     LOG [MicrosoftGetMessagesService] Messaging import for workspace e7a4c51f-4415-4cb9-b4ea-7def87c05649 and account 26efae74-5545-4d82-b0aa-05e3187c2976 fetched 200 messages in 12053ms
.
.
worker-1  | [Nest] 34  - 03/20/2025, 3:43:00 PM     LOG [MessagingMessagesImportService] Messaging import for workspace e7a4c51f-4415-4cb9-b4ea-7def87c05649 and account 26efae74-5545-4d82-b0aa-05e3187c2976 starting...
db-1      | 2025-03-20 15:43:03.177 UTC [24] LOG {ticks: 0, maint: 0, retry: 0}
db-1      | 2025-03-20 15:43:05,919 INFO: no action. I am (ccf62911b2bd), the leader with the lock
worker-1  | [Nest] 34  - 03/20/2025, 3:43:12 PM     LOG [MicrosoftGetMessagesService] Messaging import for workspace e7a4c51f-4415-4cb9-b4ea-7def87c05649 and account 26efae74-5545-4d82-b0aa-05e3187c2976 fetched 200 messages in 11385ms
.
.
worker-1  | [Nest] 34  - 03/20/2025, 3:44:00 PM     LOG [MessagingMessagesImportService] Messaging import for workspace e7a4c51f-4415-4cb9-b4ea-7def87c05649 and account 26efae74-5545-4d82-b0aa-05e3187c2976 starting...
db-1      | 2025-03-20 15:44:03.203 UTC [24] LOG {ticks: 0, maint: 0, retry: 0}
db-1      | 2025-03-20 15:44:05,920 INFO: no action. I am (ccf62911b2bd), the leader with the lock
worker-1  | Exception Captured
worker-1  |   undefined
worker-1  |   [
worker-1  |     MessageImportException [Error]: Unknown error occurred while importing messages for message channel 82b6415c-667f-4a15-b4a0-14db22e90b3c in workspace e7a4c51f-4415-4cb9-b4ea-7def87c05649: Microsoft Graph API unknown error: Error: Microsoft Graph API error: Application is over its MailboxConcurrency limit.
worker-1  |         at MessageImportExceptionHandlerService.handleUnknownException (/app/packages/twenty-server/dist/src/modules/messaging/message-import-manager/services/messaging-import-exception-handler.service.js:104:15)
worker-1  |         at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
worker-1  |         at async MessageImportExceptionHandlerService.handleDriverException (/app/packages/twenty-server/dist/src/modules/messaging/message-import-manager/services/messaging-import-exception-handler.service.js:55:17)
worker-1  |         at async MessagingMessagesImportService.processMessageBatchImport (/app/packages/twenty-server/dist/src/modules/messaging/message-import-manager/services/messaging-messages-import.service.js:122:13)
worker-1  |         at async MessagingMessagesImportJob.handle (/app/packages/twenty-server/dist/src/modules/messaging/message-import-manager/jobs/messaging-messages-import.job.js:63:9)
worker-1  |         at async MessageQueueExplorer.invokeProcessMethods (/app/packages/twenty-server/dist/src/engine/core-modules/message-queue/message-queue.explorer.js:111:17)
worker-1  |         at async MessageQueueExplorer.handleProcessor (/app/packages/twenty-server/dist/src/engine/core-modules/message-queue/message-queue.explorer.js:103:13)
worker-1  |         at async /app/packages/twenty-server/dist/src/engine/core-modules/message-queue/message-queue.explorer.js:79:17
worker-1  |         at async Worker.processFn (/app/packages/twenty-server/dist/src/engine/core-modules/message-queue/drivers/bullmq.driver.js:41:13)
worker-1  |         at async /app/node_modules/bullmq/dist/cjs/classes/worker.js:520:32 {
worker-1  |       code: 'UNKNOWN'
worker-1  |     }
worker-1  |   ]

It could have something todo with the cron job frequency? Or the batchLimit here?

Thank you for a great project! 😊

CC: @guillim

[kar-re]

Update, tried syncing again today while fiddling with cron job frequency and changing batchLimit to 4, but to no avail unfortunately.

twenty/packages/twenty-server/src/modules/messaging/message-import-manager/drivers/microsoft/services/microsoft-fetch-by-batch.service.ts

Line 23 in bd162da

const batchLimit = 20;

Seems to worked for 10 successful requests today anyway, i.e. 2000 mail imports?

[guillim]

Thanks for sharing your issue @kar-re . I am aware of a couple issues.
I fixed one last week, maybe it fixed your problem : see PR.
Working on another one this week, I will let you know once we push a fix !
🙏

[kar-re]

Thank you for the response @guillim! Unfortunately it didn't fix the issue directly, but I continued to experiment with the running container and see if I could get it to work.

It seems I could get it to work by changing export const MESSAGING_GMAIL_USERS_MESSAGES_GET_BATCH_SIZE = 200; to export const MESSAGING_GMAIL_USERS_MESSAGES_GET_BATCH_SIZE = 25;, i.e. only importing 25 messages per minute.

twenty/packages/twenty-server/src/modules/messaging/message-import-manager/drivers/gmail/constants/messaging-gmail-users-messages-get-batch-size.constant.ts

Line 1 in 3ec72a2

export const MESSAGING_GMAIL_USERS_MESSAGES_GET_BATCH_SIZE = 200;

It seems to have successfully imported around half my sent inbox, so it seems to have fixed the throttling issues. It is wierd tho, since it shouldn't send > 10 000 API requests per 10 minutes, right? As per the microsoft throttling limits. So there might be something else that accidently triggers it?

ref: Microsoft throttling limits

Thank you for your work, and lmk if you need any more debug info :)!

[guillim]

Hello,
Thanks for sharing. I think your issue is related to an issue we fixed on friday #11095 but I encountered another one that I am currently fixing. the PR is still open so not ready yet

IMO, it should fix all your problems once this is pushed

[guillim]

Closing this one since the fix was pushed and no activity since

[kar-re]

Hello! Unfortunately I think only my calendar events are imported and create contacts. But I will retry with the latest and if a bug still is there I'll make a new issue. Thank you for your work! 😊

[guillim]

Ok. Don't hesitate to continue the discussion here. I will still read and fix issues you may encounter.

[a1rwulf]

I have just configured a self-hosted instance and I also got this when testing a first sync:
twenty-worker-1 | [Nest] 34 - 07/07/2025, 8:31:06 AM ERROR [MessagingMessagesImportService] Error (TEMPORARY_ERROR) importing messages for workspace 3659418e and account 749c4962: Microsoft Graph API ApplicationThrottled 429 error: Application is over its MailboxConcurrency limit. - undefined

[anweihe]

Me too. Self hosted and connected Office365 account.