[BUG] (type-safe-api) PrepareSpecCustomResource fail with error 413 in large API with many integrations

[valebedu]

Describe the bug

When deploying an API using type-safe-api with a really large integrations object the PrepareSpecCustomResource fail with 413 because of large payload in input

Error:

1115107 byte payload is too large for the Event invocation type (limit 262144 bytes) (Service: AWSLambda; Status Code: 413; Error Code: RequestEntityTooLargeException; Request ID: 9f16fddd-3500-4106-aba5-07d6398ba67d; Proxy: null)

Expected Behavior

The PrepareSpecCustomResource process integrations successfully and deploy continue

Current Behavior

The PrepareSpecCustomResource fail and stop the deploy

Reproduction Steps

In order to reproduce the error it's pretty simple you need to make a really large integrations object and deploy this api

Here is the way I'm facing this issue in a real production scenario:

1. Follow the documentation in order to create the base type-safe-api project in TS

2. In the construct myapi.ts add the following constant and functions in order to create huge integration template responses

const RESPONSE_TEMPLATE = `#set($code = $input.body.split('<Code>')[1].split('</Code>')[0])
#set($message = $input.body.split('<Message>')[1].split('</Message>')[0])

#if($code == 'AccessControlListNotSupported')#set($errorCode = 'STOR1')
#elseif($code == 'AccessDenied')#set($errorCode = 'STOR2')
#elseif($code == 'AccessPointAlreadyOwnedByYou')#set($errorCode = 'STOR3')
#elseif($code == 'AccountProblem')#set($errorCode = 'STOR4')
#elseif($code == 'AllAccessDisabled')#set($errorCode = 'STOR5')
#elseif($code == 'AmbiguousGrantByEmailAddress')#set($errorCode = 'STOR6')
#elseif($code == 'AuthorizationHeaderMalformed')#set($errorCode = 'STOR7')
#elseif($code == 'BadDigest')#set($errorCode = 'STOR8')
#elseif($code == 'BucketAlreadyExists')#set($errorCode = 'STOR9')
#elseif($code == 'BucketAlreadyOwnedByYou')#set($errorCode = 'STOR10')
#elseif($code == 'BucketNotEmpty')#set($errorCode = 'STOR11')
#elseif($code == 'ClientTokenConflict')#set($errorCode = 'STOR12')
#elseif($code == 'CredentialsNotSupported')#set($errorCode = 'STOR13')
#elseif($code == 'CrossLocationLoggingProhibited')#set($errorCode = 'STOR14')
#elseif($code == 'EntityTooSmall')#set($errorCode = 'STOR15')
#elseif($code == 'EntityTooLarge')#set($errorCode = 'STOR16')
#elseif($code == 'ExpiredToken')#set($errorCode = 'STOR17')
#elseif($code == 'IllegalLocationConstraintException')#set($errorCode = 'STOR18')
#elseif($code == 'IllegalVersioningConfigurationException')#set($errorCode = 'STOR19')
#elseif($code == 'IncompleteBody')#set($errorCode = 'STOR20')
#elseif($code == 'IncorrectNumberOfFilesInPostRequest')#set($errorCode = 'STOR21')
#elseif($code == 'InlineDataTooLarge')#set($errorCode = 'STOR22')
#elseif($code == 'InternalError')#set($errorCode = 'STOR23')
#elseif($code == 'InvalidAccessKeyId')#set($errorCode = 'STOR24')
#elseif($code == 'InvalidAccessPoint')#set($errorCode = 'STOR25')
#elseif($code == 'InvalidAccessPointAliasError')#set($errorCode = 'STOR26')
#elseif($code == 'InvalidAddressingHeader')#set($errorCode = 'STOR27')
#elseif($code == 'InvalidArgument')#set($errorCode = 'STOR28')
#elseif($code == 'InvalidBucketAclWithObjectOwnership')#set($errorCode = 'STOR29')
#elseif($code == 'InvalidBucketName')#set($errorCode = 'STOR30')
#elseif($code == 'InvalidBucketState')#set($errorCode = 'STOR31')
#elseif($code == 'InvalidDigest')#set($errorCode = 'STOR32')
#elseif($code == 'InvalidEncryptionAlgorithmError')#set($errorCode = 'STOR33')
#elseif($code == 'InvalidLocationConstraint')#set($errorCode = 'STOR34')
#elseif($code == 'InvalidObjectState')#set($errorCode = 'STOR35')
#elseif($code == 'InvalidPart')#set($errorCode = 'STOR36')
#elseif($code == 'InvalidPartOrder')#set($errorCode = 'STOR37')
#elseif($code == 'InvalidPayer')#set($errorCode = 'STOR38')
#elseif($code == 'InvalidPolicyDocument')#set($errorCode = 'STOR39')
#elseif($code == 'InvalidRange')#set($errorCode = 'STOR40')
#elseif($code == 'InvalidRequest')#set($errorCode = 'STOR41')
#elseif($code == 'InvalidSecurity')#set($errorCode = 'STOR42')
#elseif($code == 'InvalidSOAPRequest')#set($errorCode = 'STOR43')
#elseif($code == 'InvalidStorageClass')#set($errorCode = 'STOR44')
#elseif($code == 'InvalidTargetBucketForLogging')#set($errorCode = 'STOR45')
#elseif($code == 'InvalidToken')#set($errorCode = 'STOR46')
#elseif($code == 'InvalidURI')#set($errorCode = 'STOR47')
#elseif($code == 'KeyTooLongError')#set($errorCode = 'STOR48')
#elseif($code == 'MalformedACLError')#set($errorCode = 'STOR49')
#elseif($code == 'MalformedPOSTRequest')#set($errorCode = 'STOR50')
#elseif($code == 'MalformedXML')#set($errorCode = 'STOR51')
#elseif($code == 'MaxMessageLengthExceeded')#set($errorCode = 'STOR52')
#elseif($code == 'MaxPostPreDataLengthExceededError')#set($errorCode = 'STOR53')
#elseif($code == 'MetadataTooLarge')#set($errorCode = 'STOR54')
#elseif($code == 'MethodNotAllowed')#set($errorCode = 'STOR55')
#elseif($code == 'MissingAttachment')#set($errorCode = 'STOR56')
#elseif($code == 'MissingContentLength')#set($errorCode = 'STOR57')
#elseif($code == 'MissingRequestBodyError')#set($errorCode = 'STOR58')
#elseif($code == 'MissingSecurityElement')#set($errorCode = 'STOR59')
#elseif($code == 'MissingSecurityHeader')#set($errorCode = 'STOR60')
#elseif($code == 'NoLoggingStatusForKey')#set($errorCode = 'STOR61')
#elseif($code == 'NoSuchBucket')#set($errorCode = 'STOR62')
#elseif($code == 'NoSuchBucketPolicy')#set($errorCode = 'STOR63')
#elseif($code == 'NoSuchCORSConfiguration')#set($errorCode = 'STOR64')
#elseif($code == 'NoSuchKey')#set($errorCode = 'STOR65')
#elseif($code == 'NoSuchLifecycleConfiguration')#set($errorCode = 'STOR66')
#elseif($code == 'NoSuchMultiRegionAccessPoint')#set($errorCode = 'STOR67')
#elseif($code == 'NoSuchWebsiteConfiguration')#set($errorCode = 'STOR68')
#elseif($code == 'NoSuchTagSet')#set($errorCode = 'STOR69')
#elseif($code == 'NoSuchUpload')#set($errorCode = 'STOR70')
#elseif($code == 'NoSuchVersion')#set($errorCode = 'STOR71')
#elseif($code == 'NotImplemented')#set($errorCode = 'STOR72')
#elseif($code == 'NotModified')#set($errorCode = 'STOR73')
#elseif($code == 'NotSignedUp')#set($errorCode = 'STOR74')
#elseif($code == 'OwnershipControlsNotFoundError')#set($errorCode = 'STOR75')
#elseif($code == 'OperationAborted')#set($errorCode = 'STOR76')
#elseif($code == 'PermanentRedirect')#set($errorCode = 'STOR77')
#elseif($code == 'PreconditionFailed')#set($errorCode = 'STOR78')
#elseif($code == 'Redirect')#set($errorCode = 'STOR79')
#elseif($code == 'RequestHeaderSectionTooLarge')#set($errorCode = 'STOR80')
#elseif($code == 'RequestIsNotMultiPartContent')#set($errorCode = 'STOR81')
#elseif($code == 'RequestTimeout')#set($errorCode = 'STOR82')
#elseif($code == 'RequestTimeTooSkewed')#set($errorCode = 'STOR83')
#elseif($code == 'RequestTorrentOfBucketError')#set($errorCode = 'STOR84')
#elseif($code == 'RestoreAlreadyInProgress')#set($errorCode = 'STOR85')
#elseif($code == 'ServerSideEncryptionConfigurationNotFoundError')#set($errorCode = 'STOR86')
#elseif($code == 'ServiceUnavailable')#set($errorCode = 'STOR87')
#elseif($code == 'SignatureDoesNotMatch')#set($errorCode = 'STOR88')
#elseif($code == 'SlowDown')#set($errorCode = 'STOR89')
#elseif($code == '503 SlowDown')#set($errorCode = 'STOR90')
#elseif($code == 'TemporaryRedirect')#set($errorCode = 'STOR91')
#elseif($code == 'TokenRefreshRequired')#set($errorCode = 'STOR92')
#elseif($code == 'TooManyAccessPoints')#set($errorCode = 'STOR93')
#elseif($code == 'TooManyBuckets')#set($errorCode = 'STOR94')
#elseif($code == 'TooManyMultiRegionAccessPointregionsError')#set($errorCode = 'STOR95')
#elseif($code == 'TooManyMultiRegionAccessPoints')#set($errorCode = 'STOR96')
#elseif($code == 'UnexpectedContent')#set($errorCode = 'STOR97')
#elseif($code == 'UnresolvableGrantByEmailAddress')#set($errorCode = 'STOR98')
#elseif($code == 'UserKeyMustBeSpecified')#set($errorCode = 'STOR99')
#elseif($code == 'NoSuchAccessPoint')#set($errorCode = 'STOR100')
#elseif($code == 'InvalidTag')#set($errorCode = 'STOR101')
#elseif($code == 'MalformedPolicy')#set($errorCode = 'STOR102')
#{else}#set($errorCode = 'STORX')#end

{"error": {"type": "storage_error","code": "$errorCode", "message": "$util.escapeJavaScript($message).replaceAll("\\'","'")"}, "meta": {}}`

function errorResponseSet(
  responseTemplate: string,
): typesafeapi.IntegrationResponseSet {
  return typesafeapi.IntegrationResponseSets.custom({
    responses: {
      ...Object.fromEntries(
        Array.of('400', '403', '404', '500').map((statusCode) => [
          statusCode,
          {
            statusCode: statusCode,
            responseParameters: {},
            responseTemplates: {
              'application/json': responseTemplate,
            },
          },
        ]),
      ),
      '(4|5)\\d{2}': {
        statusCode: '500',
        responseParameters: {},
        responseTemplates: {
          'application/json': responseTemplate,
        },
      },
    },
  })
}

function cacheResponseSet(
  responseTemplate: string,
): typesafeapi.IntegrationResponseSet {
  return typesafeapi.IntegrationResponseSets.composite(
    typesafeapi.IntegrationResponseSets.defaultPassthrough(),
    errorResponseSet(responseTemplate),
  )
}

3. Then in constructor create a CfnMapping in order to keep the Cloudformation Template as small as possible (this will allow to have a virtually big integrations object while keeping the cloudformation template low because it will use FindInMap)

const defaultMapping = new CfnMapping(scope, 'DefaultMapping', {
  mapping: {
    default: {
      responseTemplate: RESPONSE_TEMPLATE,
    },
  },
})

5. Then in super.integrations create a lot of integration with similar props (you'll need to create in the OpenAPI spec a lot of fake route in order to do that)

getA: {
  integration: typesafeapi.Integrations.s3({
    bucket: props.cacheBucket,
    method: 'get',
    path: 'a.json',
    integrationResponseSet: cacheResponseSet(
      defaultMapping.findInMap('default', 'responseTemplate'),
    ),
  }),
  authorizer: typesafeapi.Authorizers.none(),
},
...
getZ: {
  integration: typesafeapi.Integrations.s3({
    bucket: props.cacheBucket,
    method: 'get',
    path: 'z.json',
    integrationResponseSet: cacheResponseSet(
      defaultMapping.findInMap('default', 'responseTemplate'),
    ),
  }),
  authorizer: typesafeapi.Authorizers.none(),
},

Possible Solution

In the PrepareSpecCustomResource there are several properties:

{
  inputSpecLocation: {
    bucket: inputSpecAsset.bucket.bucketName,
    key: inputSpecAsset.s3ObjectKey,
  },
  outputSpecLocation: {
    bucket: inputSpecAsset.bucket.bucketName,
    key: preparedSpecOutputKeyPrefix,
  },
  ...prepareSpecOptions,
};

and in prepareSpecOptions integrations and everything else about the api is passed directly, so in a context of a large integrations object, the process fail everytime.

A workaround could be to write all of the info into a bucket and then read the bucket.

Additional Information/Context

No response

PDK version used

v0.23.37

What languages are you seeing this issue on?

Typescript

Environment details (OS name and version, etc.)

macOS Sonoma 14.4.1 Apple M1 Pro

[agdimech]

Hi @valebedu - thanks for raising this. I agree with your proposal - would you have capacity to raise a backwards compat PR to resolve this or would you be looking for us to add this for you? The only reason I bring this up is that we won't be able to realistically add this into the AWS PDK for another couple of weeks.

[github-actions]

This issue is now marked as stale because it hasn't seen activity for a while. Add a comment or it will be closed soon. If you wish to exclude this issue from being marked as stale, add the "backlog" label.

[github-actions]

Closing this issue as it hasn't seen activity for a while. Please add a comment @mentioning a maintainer to reopen. If you wish to exclude this issue from being marked as stale, add the "backlog" label.

[cogwirrel]

Reopening this as it's still an issue. Think it still needs a bit more digging/experimentation to figure out how we can handle large APIs without hitting the CFN/lambda size limits.