sam sync with typescript not updating Lambda layer code in Lambda function

[niklas-palm]

Description

When using sam sync for updating a Lambda Layer written in typescript, the change is not reflected in the imported code in the lambda function.

Referencing a LambdaLayer in a Lambda function, I'm expecting the lambda function to use the updated layer during invocation. This is the observed bahaviour using python, but when using a Typescript layer together with sam sync, the change is not propagated to the live function, despite the SAM cli feedback indicating it has:

Manifest is not changed for (SharedLambdaLayer), running incremental build                                                                                                                                                                                
Building layer 'SharedLambdaLayer'                                                                                                                                                                                                                        
 Running NodejsNpmBuilder:NpmPack                                                                                                                                                                                                                         
 Running NodejsNpmBuilder:CopyNpmrcAndLockfile                                                                                                                                                                                                            
 Running NodejsNpmBuilder:CopySource                                                                                                                                                                                                                      
 Running NodejsNpmBuilder:CopySource                                                                                                                                                                                                                      
 Running NodejsNpmBuilder:CleanUpNpmrc                                                                                                                                                                                                                    
Finished syncing Layer SharedLambdaLayer.                                                                                                                                                                                                                 
Syncing Function Layer Reference Sync HelloWorldFunction...                                                                                                                                                                                               
Finished syncing Function Layer Reference Sync HelloWorldFunction.      

SAM template:

  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: hello-world/
      Handler: app.lambdaHandler
      Runtime: nodejs18.x
      Architectures:
        - x86_64
      Layers:
        - !Ref SharedLambdaLayer
      ....

  SharedLambdaLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: shared-layer
      Description: Layer containing shared code
      ContentUri: shared/lib/
      RetentionPolicy: Delete
      CompatibleRuntimes:
        - nodejs18.x
    Metadata:
      BuildMethod: nodejs18.x

Steps to reproduce

(If you don't want to set it up from scratch, you can clone this repo, where I've done the below steps)

1. sam initwith a typescript hello world example.
2. Add a lambda layer in typescript, that exports a function that logs something or manipulates the return of the lambda
3. npm install inside the hello-world function diretory to install esbuild
4. sam build --build-in-source && sam deploy in the root
5. sam sync --code --watch --build-in-source in the root.
6. Update the hello-world and verify changes are live by curling the endpoint
7. Update the layer (and observe the CLI feedback says that the referenced function has been updated, after updating the layer)
8. curl the endpoint and observe that the changes have not been propagated sucessfully.

If we now make an update to the lambda function, the correct and latest layer code is used.

Observed result

Updating the Lambda layer, SAM cli syncs the layer and reports that the sync was sucessfull. Inspecting the Lambda function in the console, I also see that the layer version has been bumped. Invoking the lambda function, however, is using the old layer code.

Expected result

The latest layer code to be used.

Additional environment details

1. OS: OSX
2. 10. If using the SAM CLI, sam --version: 1.108.0
3. AWS region: eu-west-1, eu-north-1

[mndeveci]

Thanks for creating the issue, and preparing example repo for re-producing the problem.

Due to the nature of esbuild there is no way to offload some dependencies into a shared AWS::Lambda::LayerVersion. esbuild itself bundles everything into single package and that package is deployed as lambda function. For that reason, if I remove the SharedLambdaLayer and its reference in Lambda function, I can still build and invoke the function itself, even though the layer is removed from the template.

This is my updated template.yaml file;

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: >
  lambda-layer-sync-ts

  Sample SAM Template for lambda-layer-sync-ts

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 3

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: hello-world/
      Handler: app.lambdaHandler
      Runtime: nodejs18.x
      Architectures:
        - x86_64
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /hello
            Method: get
    Metadata: # Manage esbuild properties
      BuildMethod: esbuild
      BuildProperties:
        Minify: true
        Target: "es2020"
        Sourcemap: true
        EntryPoints:
          - app.ts


Outputs:
  # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
  # Find out more about other implicit resources you can reference within SAM
  # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
  HelloWorldApi:
    Description: "API Gateway endpoint URL for Prod stage for Hello World function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
  HelloWorldFunction:
    Description: "Hello World Lambda Function ARN"
    Value: !GetAtt HelloWorldFunction.Arn
  HelloWorldFunctionIamRole:
    Description: "Implicit IAM Role created for Hello World function"
    Value: !GetAtt HelloWorldFunctionRole.Arn

And I can get the sam local invoke working without any issues;

❯ sam local invoke
Invoking app.lambdaHandler (nodejs18.x)
Local image is out of date and will be updated to the latest runtime. To skip this, pass in the parameter --skip-pull-image
Building image...........................................................................................................................
Using local image: public.ecr.aws/lambda/nodejs:18-rapid-x86_64.

Mounting /private/tmp/lambda-layer-sync-ts/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated, inside runtime container
START RequestId: 37febcd6-c1cf-4838-988d-825531bfb193 Version: $LATEST
END RequestId: 34cf177a-17b1-48e4-b526-af5ac85a44a8
REPORT RequestId: 34cf177a-17b1-48e4-b526-af5ac85a44a8	Init Duration: 0.03 ms	Duration: 150.54 ms	Billed Duration: 151 ms	Memory Size: 128 MB	Max Memory Used: 128 MB
{"statusCode": 200, "body": "{\"message\":\"4 2\"}"}

However, when you remove the Layer, it might still not work since the layer contents and function contents are different folderes. sam sync process listens to file changes in the folder which is reffered by CodeUri property of the function. In order to get this work, you need to move your function and layer into a folder, and update your function definition to point to higher level folder.

• Inside template.yaml, CodeUri points to /foo
• Your lambda function code is in /foo/bar
• Your layer code is in /foo/baz

Please let us know if you have other questions or issues.

[niklas-palm]

I'm not sure I understand. In the sample I shared I am in-fact offloading shared dependencies into a layer - which is working as expected. The bug is that during sam sync, the layer is updated, which I can both manually inspect, and the logs state that the lambda functions which depend on the layers are updated. If I manually inspect the Lambda function that depends on the layer in the console, I can see that the layer it references has indeed been incremented, but the imported layer in the function itself is still the previous version, until a change to the lambda function code triggers a re-deployment and new lambda runtime.

In addition, if I'm not misstaken the change you suggest would mean that every function is re-built every time a change happens in one function. With your suggestion, my file structure would be

functions/
   function1/
      app.ts
   function2/
      app.ts
   shared/
      lib/
         utils.ts

In my template, if I set

  HelloWorldFunction1:
    Type: AWS::Serverless::Function 
    Properties:
      CodeUri: functions/
      Handler: function1.app.lambdaHandler
...

  HelloWorldFunction2:
    Type: AWS::Serverless::Function 
    Properties:
      CodeUri: functions/
      Handler: function2.app.lambdaHandler


  SharedLambdaLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: shared-layer
      Description: Layer containing shared code
      ContentUri: functions/lib/

Wouldn't that trigger all Lambdas to update if I update the code for one function?

[niklas-palm]

I just verified that is the case. I have 7 lambda function (and using the structure I mentioned in the previous message), and with the change you suggested each Lambda function is re-built during a sync whenever there's a change in one single Lambda function. I understand that all need to be re-built when the Layer code changes, but updating the code of one Lambda function then re-builds and deploys all Lambda functions.

This is not desirable and greatly impacts the development experience. The sync now takes 7x longer, since I have 7 Lambda functions.

[niklas-palm]

Would appreciate some guidance here. Given the feedback from the CLI when using sync, stating it is in fact updating functions that are referencing the layer, when the layer has been updated, this is a bug. The solution you suggested is not viable in multi-lambda projects.