Retain API Gateway permissions to old Lambda Alias/Version on update

[radahh-rest]

We are attempting to change our CDK deploy to bind the API Gateway resource to a Lambda Alias (or Version) rather than $LATEST and use Stages and Deployments to control service upgrades.

I have code that creates a version and alias and grants invoke to the alias from the APIGateway service principal. When I deploy the lambda stack, the permission for the last alias gets deleted before the stage is deployed, causing a 500 errors due to lack of permissions.

I set the retention policy of the alias to RETAIN, but it seems that the API resource addMethod function taking the new integration is what causes the change detection to remove the old permissions as the gateway resources are updated, but before the stage is deployed.

Is there a pattern I can use to retain permissions to the old version until the stage is deployed?

Here's come pseudo code showing the steps I currently have:

# One of many Lambda Stacks
        this.lambda = new LambdaFunction(this, id, {
            ...snip...config...
            currentVersionOptions: { removalPolicy: RemovalPolicy.RETAIN },
        });

        const currentVersion = this.lambda.currentVersion;
        const alias = new Alias(this, 'Alias', {
            aliasName: `live-v${currentVersion.version}`,
            version: currentVersion,
        });
        (alias.node.defaultChild as CfnResource).applyRemovalPolicy(RemovalPolicy.RETAIN);

        alias.grantInvoke(new ServicePrincipal('apigateway.amazonaws.com'));
        this.integration = new LambdaIntegration(alias);

        apiResource.addMethod(props.method, this.integration, {
            ...(props.defaultAuthorizer && { authorizer: props.defaultAuthorizer }),
        });

5 minutes later...

# Deploy Stack
        const deployment = new Deployment(this, 'Deployment', {
            api: RestApi.fromRestApiAttributes(this, 'RestApi', {
                restApiId: props.apiGateway.restApi.restApiId,
                rootResourceId: props.apiGateway.restApi.root.resourceId,
            }),
            retainDeployments: true,
        });
        deployment.addToLogicalId(new Date().valueOf());

        this.stage = new Stage(this, 'Stage', {
            deployment,
            stageName: 'prod',
            ...snip...log config...
        })

And here's the log of the lambda deploy showing the permission to the last deployed alias being removed:

endpoint-health: creating CloudFormation changeset...
endpoint-health | 0/10 | 3:46:00 pm | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack | endpoint-health User Initiated
endpoint-health | 0/10 | 3:46:04 pm | UPDATE_IN_PROGRESS   | AWS::Lambda::Function     | endpoint-health-nested-lambda/endpoint-health-nested-lambda (redacted) 
endpoint-health | 1/10 | 3:46:19 pm | UPDATE_COMPLETE      | AWS::Lambda::Function     | endpoint-health-nested-lambda/endpoint-health-nested-lambda (redacted) 
endpoint-health | 1/10 | 3:46:20 pm | CREATE_IN_PROGRESS   | AWS::Lambda::Version      | endpoint-health-nested-lambda/endpoint-health-nested-lambda/CurrentVersion (redacted)
endpoint-health | 1/10 | 3:46:21 pm | CREATE_IN_PROGRESS   | AWS::Lambda::Version      | endpoint-health-nested-lambda/endpoint-health-nested-lambda/CurrentVersion (redacted) Resource creation Initiated
endpoint-health | 2/10 | 3:46:21 pm | CREATE_COMPLETE      | AWS::Lambda::Version      | endpoint-health-nested-lambda/endpoint-health-nested-lambda/CurrentVersion (redacted)
endpoint-health | 2/10 | 3:46:24 pm | UPDATE_IN_PROGRESS   | AWS::Lambda::Alias        | endpoint-health-nested-lambda/Alias (redacted) Requested update requires the creation of a new physical resource; hence creating one.
endpoint-health | 2/10 | 3:46:24 pm | UPDATE_IN_PROGRESS   | AWS::Lambda::Alias        | endpoint-health-nested-lambda/Alias (redacted) Resource creation Initiated
endpoint-health | 3/10 | 3:46:24 pm | UPDATE_COMPLETE      | AWS::Lambda::Alias        | endpoint-health-nested-lambda/Alias (redacted)
endpoint-health | 3/10 | 3:46:27 pm | UPDATE_IN_PROGRESS   | AWS::Lambda::Permission   | endpoint-health-nested-lambda/Alias/Invokeredacted (redacted) Requested update requires the creation of a new physical resource; hence creating one.
endpoint-health | 3/10 | 3:46:27 pm | UPDATE_IN_PROGRESS   | AWS::Lambda::Permission   | endpoint-health-nested-lambda/Alias/Invokeredacted (redacted) Resource creation Initiated
endpoint-health | 4/10 | 3:46:38 pm | UPDATE_COMPLETE      | AWS::Lambda::Permission   | endpoint-health-nested-lambda/Alias/Invokeredacted (redacted)
endpoint-health | 4/10 | 3:46:40 pm | UPDATE_IN_PROGRESS   | AWS::Lambda::Permission   | endpoint-health-nested/restApi/Default/health/GET/ApiPermission.Test.redacted.GET..health (redacted) Requested update requires the creation of a new physical resource; hence creating one.
endpoint-health | 4/10 | 3:46:41 pm | UPDATE_IN_PROGRESS   | AWS::ApiGateway::Method   | endpoint-health-nested/restApi/Default/health/GET (redacted)
endpoint-health | 4/10 | 3:46:41 pm | UPDATE_IN_PROGRESS   | AWS::Lambda::Permission   | endpoint-health-nested/restApi/Default/health/GET/ApiPermission.redacted.GET..health (redacted) Requested update requires the creation of a new physical resource; hence creating one.
endpoint-health | 4/10 | 3:46:41 pm | UPDATE_IN_PROGRESS   | AWS::Lambda::Permission   | endpoint-health-nested/restApi/Default/health/GET/ApiPermission.Test.redacted.GET..health (redacted) Resource creation Initiated
endpoint-health | 4/10 | 3:46:41 pm | UPDATE_IN_PROGRESS   | AWS::Lambda::Permission   | endpoint-health-nested/restApi/Default/health/GET/ApiPermission.redacted.GET..health (redacted) Resource creation Initiated
endpoint-health | 5/10 | 3:46:41 pm | UPDATE_COMPLETE      | AWS::ApiGateway::Method   | endpoint-health-nested/restApi/Default/health/GET (redacted) 
endpoint-health | 6/10 | 3:46:51 pm | UPDATE_COMPLETE      | AWS::Lambda::Permission   | endpoint-health-nested/restApi/Default/health/GET/ApiPermission.Test.redacted.GET..health (redacted)
endpoint-health | 7/10 | 3:46:51 pm | UPDATE_COMPLETE      | AWS::Lambda::Permission   | endpoint-health-nested/restApi/Default/health/GET/ApiPermission.redacted.GET..health (redacted)
endpoint-health | 8/10 | 3:46:53 pm | UPDATE_COMPLETE_CLEA | AWS::CloudFormation::Stack | endpoint-health 
endpoint-health | 8/10 | 3:46:54 pm | DELETE_IN_PROGRESS   | AWS::Lambda::Permission   | endpoint-health-nested/restApi/Default/health/GET/ApiPermission.Test.redacted.GET..health (redacted)
endpoint-health | 8/10 | 3:46:54 pm | DELETE_IN_PROGRESS   | AWS::Lambda::Permission   | endpoint-health-nested/restApi/Default/health/GET/ApiPermission.redacted.GET..health (redacted)
endpoint-health | 7/10 | 3:47:04 pm | DELETE_COMPLETE      | AWS::Lambda::Permission   | endpoint-health-nested/restApi/Default/health/GET/ApiPermission.Test.redacted.GET..health (redacted)
endpoint-health | 6/10 | 3:47:04 pm | DELETE_COMPLETE      | AWS::Lambda::Permission   | endpoint-health-nested/restApi/Default/health/GET/ApiPermission.redacted.GET..health (redacted)
endpoint-health | 6/10 | 3:47:05 pm | DELETE_IN_PROGRESS   | AWS::Lambda::Permission   | endpoint-health-nested-lambda/Alias/Invokeredacted (redacted)
endpoint-health | 5/10 | 3:47:15 pm | DELETE_COMPLETE      | AWS::Lambda::Permission   | endpoint-health-nested-lambda/Alias/Invokeredacted (redacted)
endpoint-health | 5/10 | 3:47:16 pm | DELETE_SKIPPED       | AWS::Lambda::Alias        | endpoint-health-nested-lambda/Alias (redacted) 
endpoint-health | 5/10 | 3:47:17 pm | DELETE_SKIPPED       | AWS::Lambda::Version      | endpointhealthnestedlambdaCurrentVersion7D9redacted
endpoint-health | 6/10 | 3:47:17 pm | UPDATE_COMPLETE      | AWS::CloudFormation::Stack | endpoint-health

[radahh-rest]

I created broader permissions and can see that the update goes live immediately, before the stage is deployed. I'm guessing then that you must use stage variables to achieve this.