Bug: sam sync does not update layer version in dependent lambdas

[maksymdukov]

Description:

sam sync --watch and --code does not sync Function Layer Reference under certain conditions when Layer itself gets synced.

Steps to reproduce:

Create serverless template yaml and put it in <rootDir>/cf/template.yaml

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: Sam sync

Globals:
  Function:
    Layers:
      - !Ref LambdaLayer

Resources:
  LambdaLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: !Sub my-layer-${AWS::StackName}
      Description: Common Library
      ContentUri: ../src/common/
      CompatibleRuntimes:
        - nodejs14.x
    Metadata:
      BuildMethod: makefile

  LambdaFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: !Sub test-sam-sync-${AWS::StackName}
      CodeUri: ../src/handlers
      Handler: handler.handler
      Runtime: nodejs14.x
      MemorySize: 128

<rootDir>/src/common/Makefile

build-LambdaLayer:
	cp -r ./ "$(ARTIFACTS_DIR)"

<rootDir>/src/common/log.js

module.exports = {
  log: () => {
    console.log('dummy');
  },
};

<rootDir>/src/handlers/handle.js

const { log } = require('/opt/log');

exports.handler = async (event, context) => {
  log();
};

From the rooDir run sam sync:

sam sync \
    -t ./cf/template.yaml \
    --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND \
    --stack-name sam-sync-test \
    --watch

Try to change code in the layer <rootDir>/src/common/log.js

Observed result:

Updated layer gets synced but Lambda's Layer Reference does not get updated, meaning Lambda still points to the old layer version

sam sync \
    -t ./cf/template.yaml \
    --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND \
    --stack-name sam-sync-test \
    --s3-bucket sam-sync-test \
    --s3-prefix sam-sync \
    --watch
    --debug

The SAM CLI will use the AWS Lambda, Amazon API Gateway, and AWS StepFunctions APIs to upload your code without 
performing a CloudFormation deployment. This will cause drift in your CloudFormation stack. 
**The sync command should only be used against a development stack**.
Confirm that you are synchronizing a development stack.

Enter Y to proceed with the command, or enter N to cancel:
 [Y/n]: y
Queued infra sync. Wating for in progress code syncs to complete...
Starting infra sync.
Valid cache found, copying previously built resources for following layers (LambdaLayer)
Building codeuri: /home/max/study/aws/maksym.dukov/sam-data-processing/cf runtime: nodejs14.x metadata: {} architecture: x86_64 functions: LambdaFunction
package.json file not found. Continuing the build without dependencies.
Running NodejsNpmBuilder:CopySource

Build Succeeded

Successfully packaged artifacts and wrote output template to file /tmp/tmpm8fplfsb.
Execute the following command to deploy the packaged template
sam deploy --template-file /tmp/tmpm8fplfsb --stack-name <YOUR STACK NAME>


        Deploying with following values
        ===============================
        Stack name                   : sam-sync-test
        Region                       : us-east-1
        Disable rollback             : False
        Deployment s3 bucket         : sam-sync-test
        Capabilities                 : ["CAPABILITY_IAM", "CAPABILITY_NAMED_IAM", "CAPABILITY_AUTO_EXPAND"]
        Parameter overrides          : {}
        Signing Profiles             : null

Initiating deployment
=====================

2022-07-23 22:08:25 - Waiting for stack create/update to complete

CloudFormation events from stack operations (refresh every 0.5 seconds)
---------------------------------------------------------------------------------------------------------------------
ResourceStatus                ResourceType                  LogicalResourceId             ResourceStatusReason        
---------------------------------------------------------------------------------------------------------------------
UPDATE_IN_PROGRESS            AWS::CloudFormation::Stack    sam-sync-test                 Transformation succeeded    
UPDATE_IN_PROGRESS            AWS::CloudFormation::Stack    AwsSamAutoDependencyLayerNe   -                           
                                                            stedStack                                                 
UPDATE_COMPLETE               AWS::CloudFormation::Stack    AwsSamAutoDependencyLayerNe   -                           
                                                            stedStack                                                 
UPDATE_COMPLETE_CLEANUP_IN_   AWS::CloudFormation::Stack    sam-sync-test                 -                           
PROGRESS                                                                                                              
UPDATE_COMPLETE               AWS::CloudFormation::Stack    AwsSamAutoDependencyLayerNe   -                           
                                                            stedStack                                                 
UPDATE_COMPLETE               AWS::CloudFormation::Stack    sam-sync-test                 -                           
---------------------------------------------------------------------------------------------------------------------

Stack update succeeded. Sync infra completed.

Infra sync completed.
Syncing Layer LambdaLayer...
Cache is invalid, running build and copying resources for following layers (LambdaLayer)
Building layer 'LambdaLayer'
Running CustomMakeBuilder:CopySource
Running CustomMakeBuilder:MakeBuild
Current Artifacts Directory : /home/max/study/aws/maksym.dukov/sam-data-processing/.aws-sam/auto-dependency-layer/LambdaLayer
cp -r ./ "/home/max/study/aws/maksym.dukov/sam-data-processing/.aws-sam/auto-dependency-layer/LambdaLayer"
Updated source_hash and manifest_hash field in build.toml for layer with UUID 432944d3-6c2a-4123-866e-6ef7c6acef96
Finished syncing Layer LambdaLayer.

Expected result:

Lambda's Layer Reference gets updated

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

Workaround (1) to make it all work - specify - !Ref LambdaLayer in a lambda declaration section instead of the Globals section

Workaround (2) - move template.yaml into the root directory.

1. OS: ManjaroLinux 21.3.5
2. sam --version: SAM CLI, version 1.53.0
3. AWS region: us-east-1

Add --debug flag to command you are running

[qingchm]

Thanks for reporting this in! We will first try to reproduce this issue using information you provided

[qingchm]

@maksymdukov Sorry for a postponed response here, as I look at the issue carefully I wonder if we are missing some logic to create function layer reference syncs when the layer attribute is defined in functions. Can I confirm with you that this only happens when you define the layer reference in globals? Meanwhile we will reproduce the issue and investigate if the get dependent functions method didn't return all functions for your case

Below is how we are finding the layer's related functions:

    def _get_dependent_functions(self) -> List[Function]:
        function_provider = SamFunctionProvider(cast(List[Stack], self._stacks), locate_layer_nested=True)

        dependent_functions = []
        for function in function_provider.get_all():
            if self._layer_identifier in [layer.full_path for layer in function.layers]:
                LOG.debug(
                    "%sAdding function %s for updating its Layers with this new version",
                    self.log_prefix,
                    function.name,
                )
                dependent_functions.append(function)
        return dependent_functions

[qingchm]

Also just assigning myself here to keep track of the issue

[RobertMarton1985]

We are also impacted by this issue. The issue happens only when the layer is defined in the Globals section.

[wynged]

Just ran into this issue as well. Would love any thoughts about timing to resolve. we're thinking about giving up using Globals for this because of how much worse it makes sam sync

[NateOelke]

Similarly, I am using nested templates where the layers are defined in one template and the functions that use the layers are in a different template. The deployments work correctly, but when I use sam sync --watch and update the layer code, the layers are updated but the functions are not updated to point to the latest layer. Are there any known workarounds to get sam sync working properly with nested templates?

Here is how I'm currently exporting the layer reference:

Outputs:
  LayerArn:
    Value: !Ref Layer
    Export:
      Name: LayerArn

Here is how I'm using the layer reference:

  Function:
    ...
    Layers:
      - !ImportValue LayerArn

[tw-Dan]

This still seems to be an issue.