[BUG] Static Website deployments fail due to throttling when WebACL configuration is included

[wawajo]

Describe the bug

When deploying a stack including the StaticWebsite construct and passing webAclProps to define a cidrAllowList, deployments fail due to throttling (100% of attempts in my current test stack).

Expected Behavior

Stack deploys successfully.

Current Behavior

Stack deployment fails with "Rate exceeded" while attempting to run the custom resource to stand up a WebACL.

demo-stack-test | 29/38 | 9:14:04 AM | CREATE_FAILED        | AWS::CloudFormation::CustomResource             | DemoWebsite/DemoWebsite/WebsiteAcl/CFAclCustomResource/Default (DemoWebsiteWebsiteAclCFAclCustomResource6EA5002F) Received response status [FAILED] from custom resource. Message returned: Rate exceeded

Logs: /aws/lambda/demo-stack-test-AclEvent-39e515

    at throwDefaultError (/var/runtime/node_modules/@aws-sdk/node_modules/@smithy/smithy-client/dist-cjs/index.js:838:20)
    at /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/smithy-client/dist-cjs/index.js:847:5
    at de_CommandError (/var/runtime/node_modules/@aws-sdk/client-wafv2/dist-cjs/index.js:2208:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
    at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/core/dist-cjs/index.js:165:18
    at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
    at async /var/runtime/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22
    at async createWafAcl (/var/task/index.js:119:42)
    at async createWaf (/var/task/index.js:104:34) (RequestId: 6e1b4fb5-b09f-42f8-9dca-2c57d7ca0e6e)

Reproduction Steps

export class WebsiteConstruct extends Construct {
  constructor(scope: Construct, id: string, props?: WebsiteConstructProps) {
    super(scope, id);

    new StaticWebsite(this, id, {
      websiteContentPath: "../demo-ui/out",
      runtimeOptions: {
        jsonPayload: {
          region: Stack.of(this).region,
          identityPoolId: props?.userIdentity.identityPool.identityPoolId,
          userPoolId: props?.userIdentity.userPool?.userPoolId,
          userPoolWebClientId:
            props?.userIdentity.userPoolClient?.userPoolClientId,
        },
      },
      distributionProps: {
        geoRestriction: GeoRestriction.allowlist(
          "AU",
          "ID",
          "IN",
          "JP",
          "KR",
          "SG",
          "US",
        ),
      },
      webAclProps: {
        cidrAllowList: {
          cidrType: "IPV4" as const,
            cidrRanges: [
              // IP ranges here
           ],
         },
      },
    });
  }
}

Possible Solution

Issue seems to have been introduced with this fix: #711

Signs point to the current retry logic being overly aggressive combined with lack of backoff.

It's possible that throttling thresholds for the relevant APIs were tightened after the initial fix was committed or that the issue only appears in combination with some other details of the deployment.

The simplest fix that worked for me in testing was to increase the RETRY_INTERVAL introduced as part of the fix.

i.e.

const MAX_CREATE_RETRY = 10;
const RETRY_INTERVAL = 200;

changed to

const MAX_CREATE_RETRY = 10;
const RETRY_INTERVAL = 2000;

Additional Information/Context

No response

PDK version used

0.23.10

What languages are you seeing this issue on?

Typescript

Environment details (OS name and version, etc.)

MacOS 13.6.3