Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[lambda] deployment failure on updates to cross-stack layers #1972

Open
nchorniy opened this issue Mar 7, 2019 · 13 comments
Open

[lambda] deployment failure on updates to cross-stack layers #1972

nchorniy opened this issue Mar 7, 2019 · 13 comments
Assignees

Comments

@nchorniy
Copy link

@nchorniy nchorniy commented Mar 7, 2019

Here is the application structure:

  • Stack A:
    • lambda layer
  • Stack B: Depends on A:
    • Lambda which is using the layer
new lambda.Function(this, 'some-lambda'
   {
     ...
        layers: [stackA.lambdaLayer]
   }
)

where stackA.lambdaLayer is an instance of lambda.LayerVersion.

The initial deployment works well.
However, when the layer code is changed and the second deployment takes place, Stack A deployment fails with error message:

lambdas-layer (...) Requested update requires the creation of a new physical resource; hence
creating one.
Resource creation Initiated
dev-StackA Export dev-CoreStack:corelambdaslayerLayerVersionArn261FB3DB
cannot be updated as it is in use by dev-StackB

@lapair

This comment has been minimized.

Copy link

@lapair lapair commented Jul 10, 2019

@nchorniy is this still broken out of interest?... I was about to try set it up, but thought I'd just ask instead :)

@hughevans hughevans mentioned this issue Aug 9, 2019
1 of 1 task complete
@alexdilley

This comment has been minimized.

Copy link

@alexdilley alexdilley commented Aug 15, 2019

@lapair still broken; I have the same issue.

@eladb

This comment has been minimized.

Copy link
Contributor

@eladb eladb commented Aug 15, 2019

Okay, I see what's going on. I don't have a solution yet, let me first explain what causes this: the problem here is that the layer version ARN changes when you update the layer (makes sense so far), which causes a change in the exported value which is imported from Stack B. CloudFormation protects against changes in export values that are being used by other stacks in order to protect against breaking the depending stack.

@eladb eladb self-assigned this Aug 15, 2019
@alexdilley

This comment has been minimized.

Copy link

@alexdilley alexdilley commented Aug 15, 2019

@eladb I've worked around this by using a SSM parameter store to reference the latest
layerVersionArn and then accessing the layer instance via Layer.fromLayerVersionArn(). (Nod to @rhboyd for this idea.)

I wrote a custom construct to act as such a proxy:

const cdk = require('@aws-cdk/core');
const lambda = require('@aws-cdk/aws-lambda');
const ssm = require('@aws-cdk/aws-ssm');

// Proxy to the LATEST version of the Lambda Layer shared between stacks via a
// SSM paramter store.
module.exports = class SharedLayer extends cdk.Construct {
  static parameterName(construct) {
    return `/${cdk.Stack.of(construct).stage}/shared/layerVersionArn`;
  }

  static of(construct) {
    const layerVersionArn = ssm.StringParameter.valueForStringParameter(
      construct,
      this.parameterName(construct)
    );

    return lambda.LayerVersion.fromLayerVersionArn(
      construct,
      'SharedLayer',
      layerVersionArn
    );
  }

  constructor(scope, id, props) {
    super(scope, id, props);

    new ssm.StringParameter(this, 'VersionArn', {
      parameterName: SharedLayer.parameterName(this),
      stringValue: props.layerVersionArn,
    });
  }
};
@NGL321 NGL321 added guidance and removed question labels Aug 23, 2019
@jogold

This comment has been minimized.

Copy link
Contributor

@jogold jogold commented Aug 23, 2019

Okay, I see what's going on. I don't have a solution yet,

@eladb how about allowing the users to specify how they want to resolve their cross-stacks references: either with Fn::Import (the normal case) or with a custom resource that will read the output value by calling the CloudFormation API. This would allow to have cross-stacks references for resources for which an update sometimes requires a replacement.

@jogold

This comment has been minimized.

Copy link
Contributor

@jogold jogold commented Aug 26, 2019

This would result in a downtime for the consuming stack but can be maybe mitigated with a DeletionPolicy in the producing stack together with some sort of clean up at the end of the deploy command...

@ranguard

This comment has been minimized.

Copy link
Contributor

@ranguard ranguard commented Aug 26, 2019

@eladb eladb assigned nija-at and unassigned eladb Sep 3, 2019
@fabio-vitali

This comment has been minimized.

Copy link

@fabio-vitali fabio-vitali commented Sep 9, 2019

@alexdilley could you post a sample code showing how your construct solves the scenario described in this issue?

@rhboyd

This comment has been minimized.

Copy link
Contributor

@rhboyd rhboyd commented Sep 9, 2019

@fabio-vitali That solutions works by breaking the coupling between one stack's export value and another stack's import value (which is enforced by cloudformation). Instead they are using one stack to put a value into SSM Parameter store then another stack is reading values from the same SSM Parameter store.

@lapair

This comment has been minimized.

Copy link

@lapair lapair commented Sep 17, 2019

@fabio-vitali

The code @alexdilley wrote breaks down into:

Creating layer

        const paramName = '/layers/BaseLayer';

        // Create Layer
        const baseLayer = new lambda.LayerVersion(this, 'BaseLayer', {
            code: lambda.Code.asset(__dirname + '/layer/base'),
            compatibleRuntimes: [
                lambda.Runtime.NODEJS_8_10
            ],
            description: 'Our base layer',
        });

        // Save to SSM
        new ssm.StringParameter(this, 'VersionArn', {
            parameterName: paramName,
            stringValue: baseLayer.layerVersionArn,
        });

Using layer:

        const paramName = '/layers/BaseLayer';


         // fetch the Arn from param store
        const baseLayerArn = ssm.StringParameter.valueForStringParameter(
            this,
            paramName
        );
       // generate layerversion from arn 
        const layer1 = lambda.LayerVersion.fromLayerVersionArn(
            this,
            'BaseLayerFromArn',
            baseLayerArn
        );

       // Then supply when you create a lambda
       new lambda.Function(this, 'SomeName, {
              ...,
             layers: [layer1],
      });
@lapair

This comment has been minimized.

Copy link

@lapair lapair commented Sep 17, 2019

@fabio-vitali but I think the @alexdilley version might create the relationship that my example above does not? Either way.. updating the layer deletes the previous version (with CDK).... but...

"To avoid breaking functions, a copy of the version remains in Lambda until no functions refer to it."
https://docs.aws.amazon.com/cli/latest/reference/lambda/delete-layer-version.html

@chrisgel15

This comment has been minimized.

Copy link

@chrisgel15 chrisgel15 commented Sep 19, 2019

Not using CDK but I have been affected by this CF issue as well. My stack is created via CodePipeline + CloudFormation templates and I have a LayerVersion resource declared on it.

When I used the Output section to export my layer, the first time worked correctly but the second one showed the same error message.

I ended up declaring my LayerVersion resource in both templates to avoid using the cross reference value, as everytime the pipeline runs, it creates a new LayerVersion (no matter if I set the RetentionPolicy to Delete), and that version modification is preventing the update of the 2nd stack.

@pmaedel

This comment has been minimized.

Copy link

@pmaedel pmaedel commented Oct 22, 2019

see #4014 for a feature request that would solve this issue

@SomayaB SomayaB added the in-progress label Nov 1, 2019
@SomayaB SomayaB added bug and removed guidance labels Nov 11, 2019
@nija-at nija-at added p2 and removed in-progress labels Nov 12, 2019
@nija-at nija-at changed the title Lambda Layer redeployment fails [lambda] Deployment failure on updates to cross-stack layers Nov 12, 2019
@nija-at nija-at changed the title [lambda] Deployment failure on updates to cross-stack layers [lambda] deployment failure on updates to cross-stack layers Nov 12, 2019
shaftoe added a commit to shaftoe/api-l3x-in that referenced this issue Mar 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.