[BUG][Functional] Throttling issues with exponential-backoff. #1023
Description
Bug reports which fail to provide the required information will be closed without action.
Required Basic Info
- Accelerator Version: Discovered in 1.5.0, believe code is affected in all versions.
Describe the bug
While running the accelerator to suspend five accounts in a client environment, we encountered TooManyRequestsException in Accel-Security-Phase2 when calling MacieCreateMember. Working with the Macie team, from their data it appears that the throttlingBackOff is not applying jitter to the calls and the current setting for startingDelay is too low, which was causing the Macie service to throttle. Re-running the accelerator may resolve the 429/throttling, but for larger numbers of account changes, the more likely the threshold will be hit.
Failure Info
Failed resources:
From a previous run:
PBMMAccel-Security-Phase2 | 5:35:11 PM | UPDATE_FAILED | Custom::MacieCreateMember | SecurityPhase2ApSoutheast_2/MacieCreateMember31/Resource/Default (MacieCreateMember3134C1F2B8) Received response status [FAILED] from custom resource. Message returned: TooManyRequestsException: Too Many Requests (RequestId: 2a83e3fd-cc4d-4d25-a90e-25c4128d4cf3)
new CustomResource (/app/node_modules/.pnpm/@aws-cdk+core@1.144.0/node_modules/@aws-cdk/core/lib/custom-resource.ts:36:21)
_ new MacieCreateMember (/app/src/lib/custom-resources/cdk-macie-create-member/cdk/index.ts:40:21)
_ /app/src/deployments/cdk/src/deployments/macie/macie.ts:205:25
_ Array.map ()
_ Object.step2 (/app/src/deployments/cdk/src/deployments/macie/macie.ts:187:11)
_ processTicksAndRejections (internal/process/task_queues.js:95:5)
_ deploy (/app/src/deployments/cdk/src/apps/phase-2.ts:408:3)
_ Object.deploy (/app/src/deployments/cdk/src/app.ts:87:3)
_ main (/app/src/deployments/cdk/cdk.ts:92:16)
Required files
Not applicable.
Steps To Reproduce
Problem occurs intermittently.
Expected behavior
No 429/throttling when creating or suspending accounts.
Screenshots
Additional context