Breaking change introduced for Custom Authorizer Lambdas

[DrColza]

Bug Report

Merged PR 38f6ac1 introduced breaking changes for lambda based custom authorizer.

Description

Given I am running serverless v1.60.5
When I run sls deploy
Then the deploy fails because the "cloudformation-template-update-stack.json" file has an incorrect datatype in place for the FunctionName of the Authorizerr Function
Error: An error occurred: CustomAuthorizerLambdaPermissionApiGateway - Value of property FunctionName must be of type String.

Source code that changed https://github.com/serverless/serverless/blob/38f6ac125e54d927871b4e5f5b387e0d4c28a6a7/lib/plugins/aws/package/compile/events/apiGateway/lib/permissions.js commit #38f6ac1

Resultant Cloudformation code showing that FunctionName isn't a String variable:

"CustomAuthorizerLambdaPermissionApiGateway": {
      "Type": "AWS::Lambda::Permission",
      "Properties": {
        "FunctionName": {
          "Fn::Join": [
            ":",
            [
              {
                "Fn::GetAtt": [
                  "CustomAuthorizerLambdaFunction",
                  "Arn"
                ]
              }
            ]
          ],
          "Fn::GetAtt": [
            "CustomAuthorizerLambdaFunction",
            "Arn"
          ]
        },

1. What did you do?
   Updated serverless from v1.59.3 to v1.60.5
2. What happened?
   Deploys started to fail because the CustomAuthorizer function could not be deployed
3. What should've happened?
   It should have delpoyed, like it previously had done for the past 18 months.
4. What's the content of your serverless.yml file?
   Pasting in the relevant data:

frameworkVersion: ">=1.1.0 <2.0.0"

provider:
  name: aws
  runtime: python3.7
  stage: ${opt:stage,'dev'}
  region: eu-west-1
  tracing:
    lambda: true
    apiGateway: true
plugins:
  - serverless-python-requirements
  - serverless-stage-manager
  - serverless-prune-plugin

functions:
  customAuthorizer:
    handler: authorizer.auth
    events:
        - http:
            path: /auth
            method: GET
            cors: true

1. What's the output you get when you use the SLS_DEBUG=* environment variable (e.g. SLS_DEBUG=* serverless deploy)
   Serverless Error ---------------------------------------

An error occurred: CustomAuthorizerLambdaPermissionApiGateway - Value of property FunctionName must be of type String.

Get Support --------------------------------------------
Docs: docs.serverless.com
Bugs: github.com/serverless/serverless/issues
Issues: forum.serverless.com

Your Environment Information ---------------------------
Operating System: darwin
Node Version: 11.13.0
Framework Version: 1.60.5
Plugin Version: 3.2.7
SDK Version: 2.2.1
Components Core Version: 1.1.2
Components CLI Version: 1.4.0

Similar or dependent issues:
The change to the relevant code was introduced as a result of the PR raised from issue:

• Provisioned Concurrency Does Not Work As Described #7059
  

  serverless/lib/plugins/aws/package/compile/events/apiGateway/lib/permissions.js

  Lines 19 to 29 in 38f6ac1

  	FunctionName: {
  	'Fn::Join': [
  	':',
  	[
  	{
  	'Fn::GetAtt': [lambdaLogicalId, 'Arn'],
  	},
  	...(lambdaAliasName ? [lambdaAliasName] : []),
  	],
  	],
  	},

[medikoo]

Fn::Join is a valid CloudFormation instruction, and this change is confirmed to work.

I'm guessing that some plugin relied on specific format of FunctionName, hence the error. Can you run same with SLS_DEBUG=* and check where exactly crash happens?

[DrColza]

The Fn::Join is only joining ":" and [{"Fn::GetAtt": ["CustomAuthorizerLambdaFunction","Arn"]}]

The "Fn::GetAtt": ["CustomAuthorizerLambdaFunction","Arn"] lies outside of the closing square bracket for Fn::Join

"Fn::Join": [
                ":", 
                [{"Fn::GetAtt": ["CustomAuthorizerLambdaFunction","Arn"]}]
            ],
"Fn::GetAtt": ["CustomAuthorizerLambdaFunction","Arn"]

That's where I thought the issue was coming from.

Here is the error output from a deploy:

Serverless: View the full error output: https://eu-west-1.console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stack/detail?stackId=arn%3Aaws%3Acloudformation%3Aeu-west-1%3A389957958676%3Astack%2Fcmp-api-dev%2F47694580-7c9a-11e9-8bae-0a62311bc72a

  Serverless Error ---------------------------------------

  ServerlessError: An error occurred: CustomAuthorizerLambdaPermissionApiGateway - Value of property FunctionName must be of type String.
      at provider.request.then.data (/usr/local/lib/node_modules/serverless/lib/plugins/aws/lib/monitorStack.js:125:33)
  From previous event:
      at AwsDeploy.monitorStack (/usr/local/lib/node_modules/serverless/lib/plugins/aws/lib/monitorStack.js:28:12)
      at provider.request.then.cfData (/usr/local/lib/node_modules/serverless/lib/plugins/aws/lib/updateStack.js:103:28)
  From previous event:
      at AwsDeploy.update (/usr/local/lib/node_modules/serverless/lib/plugins/aws/lib/updateStack.js:103:8)
  From previous event:
      at AwsDeploy.BbPromise.bind.then (/usr/local/lib/node_modules/serverless/lib/plugins/aws/lib/updateStack.js:117:35)
  From previous event:
      at AwsDeploy.updateStack (/usr/local/lib/node_modules/serverless/lib/plugins/aws/lib/updateStack.js:113:33)
  From previous event:
      at AwsDeploy.BbPromise.bind.then (/usr/local/lib/node_modules/serverless/lib/plugins/aws/deploy/index.js:127:39)
  From previous event:
      at Object.aws:deploy:deploy:updateStack [as hook] (/usr/local/lib/node_modules/serverless/lib/plugins/aws/deploy/index.js:123:30)
      at BbPromise.reduce (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:490:55)
  From previous event:
      at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:490:22)
      at PluginManager.spawn (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:510:17)
      at AwsDeploy.BbPromise.bind.then (/usr/local/lib/node_modules/serverless/lib/plugins/aws/deploy/index.js:93:48)
  From previous event:
      at Object.deploy:deploy [as hook] (/usr/local/lib/node_modules/serverless/lib/plugins/aws/deploy/index.js:89:30)
      at BbPromise.reduce (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:490:55)
  From previous event:
      at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:490:22)
      at getHooks.reduce.then (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:525:24)
  From previous event:
      at PluginManager.run (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:525:8)
      at variables.populateService.then (/usr/local/lib/node_modules/serverless/lib/Serverless.js:133:33)
      at processImmediate (internal/timers.js:443:21)
      at process.topLevelDomainCallback (domain.js:136:23)
  From previous event:
      at Serverless.run (/usr/local/lib/node_modules/serverless/lib/Serverless.js:120:74)
      at serverless.init.then (/usr/local/lib/node_modules/serverless/bin/serverless.js:75:30)
      at /usr/local/lib/node_modules/serverless/node_modules/graceful-fs/graceful-fs.js:111:16
      at /usr/local/lib/node_modules/serverless/node_modules/graceful-fs/graceful-fs.js:45:10
      at FSReqCallback.args [as oncomplete] (fs.js:145:20)
  From previous event:
      at initializeErrorReporter.then (/usr/local/lib/node_modules/serverless/bin/serverless.js:75:8)
      at processImmediate (internal/timers.js:443:21)
      at process.topLevelDomainCallback (domain.js:136:23)
  From previous event:
      at Object.<anonymous> (/usr/local/lib/node_modules/serverless/bin/serverless.js:64:4)
      at Module._compile (internal/modules/cjs/loader.js:805:30)
      at Object.Module._extensions..js (internal/modules/cjs/loader.js:816:10)
      at Module.load (internal/modules/cjs/loader.js:672:32)
      at tryModuleLoad (internal/modules/cjs/loader.js:612:12)
      at Function.Module._load (internal/modules/cjs/loader.js:604:3)
      at Function.Module.runMain (internal/modules/cjs/loader.js:868:12)
      at internal/main/run_main_module.js:21:11

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information ---------------------------
     Operating System:          darwin
     Node Version:              11.13.0
     Framework Version:         1.60.5
     Plugin Version:            3.2.7
     SDK Version:               2.2.1
     Components Core Version:   1.1.2
     Components CLI Version:    1.4.0

[DrColza]

P.S Thanks for the quick reply 👍

[medikoo]

@DrColza can you check what is the template as generated by framework without plugins involved.

I believe framework produces a valid template and it's one of the plugins which you rely on that introduces breaking changes to it

[DrColza]

Given I comment out the plugins section in serverless.yml
When I run sls package --stage dev
Then I get the following JSON in cloudformation-template-update-stack.json

"CustomAuthorizerLambdaPermissionApiGateway": {
      "Type": "AWS::Lambda::Permission",
      "Properties": {
        "FunctionName": {
          "Fn::Join": [
            ":",
            [
              {
                "Fn::GetAtt": [
                  "CustomAuthorizerLambdaFunction",
                  "Arn"
                ]
              }
            ]
          ],
          "Fn::GetAtt": [
            "CustomAuthorizerLambdaFunction",
            "Arn"
          ]
        },

When I run the same scenario in v1.59.3
Then I get the follwoing JSON:

"CustomAuthorizerLambdaPermissionApiGateway": {
      "Type": "AWS::Lambda::Permission",
      "Properties": {
        "FunctionName": {
          "Fn::GetAtt": [
            "CustomAuthorizerLambdaFunction",
            "Arn"
          ]
        },

[medikoo]

@DrColza I'm not able to reproduce that. I've created very similar service as yours:
https://github.com/medikoo/test-serverless/blob/7189-case/serverless.yml

And it deploys successfully with v1.60.5, also we can see that AWS::Lambda::Permission resource is generated as expected: https://github.com/medikoo/test-serverless/blob/f0874b530b82f916307b771ee18636c29fa349d1/.serverless/cloudformation-template-update-stack.json#L342-L354

The only difference is that it's for node12.x runtime. Still I've switched to Python, tested packaging, and shape of AWS::Lambda::Permission was not affected.

If you're positive you get that without any plugins involved, can you update this test case, so i can reproduce issue on my side (?)

[DrColza]

Hi @medikoo

I can replicate it with this (in Nodejs) 1.60.5

provider:
  name: aws
  runtime: nodejs12.x
  region: us-east-1
  stage: dev
  tracing:
    lambda: true
    apiGateway: true

functions:
  customAuthorizer:
    handler: index.auth
    events:
      - http:
          path: /auth
          method: GET
          cors: true

  function:
    handler: index.handler
    events:
      - http:
          path: foo
          method: GET
          cors: true
          authorizer: customAuthorizer

      "Type": "AWS::Lambda::Permission",
      "Properties": {
        "FunctionName": {
          "Fn::Join": [
            ":",
            [
              {
                "Fn::GetAtt": [
                  "CustomAuthorizerLambdaFunction",
                  "Arn"
                ]
              }
            ]
          ],
          "Fn::GetAtt": [
            "CustomAuthorizerLambdaFunction",
            "Arn"
          ]
        },```

[medikoo]

Thanks @DrColza, I see it now, it should be fixed with #7197.

Can you confirm that this patch fixes the issue on your side (not only deployment, but that authorizer also works without issues)?

You can install patched version by referencing it via serverless@serverless/serverless#0109-fix-permissions-setup

[DrColza]

Hi @medikoo I can confirm the deploy works as expected however the custom Authorizer is not working correctly anymore.

sls package
results in:

      "Type": "AWS::Lambda::Permission",
      "Properties": {
        "FunctionName": {
          "Fn::GetAtt": [
            "CustomAuthorizerLambdaFunction",
            "Arn"
          ]
        },

I'm troubleshooting to see what the source of the issue is, whether it's caused by serverless itself or a plugin.

The issue is that a read-only policy (Only permits invoking GET endpoints) is being returned from the customAuthorizer function but access to Write endpoints (POST methods) is still being permitted.

The code for the customAuthorizer function hasn't changed and is working in Production but when I deployed to Dev using serverless@serverless/serverless#0109-fix-permissions-setup I'm getting the above permissions error.

[medikoo]

I'm troubleshooting to see what the source of the issue is, whether it's caused by serverless itself or a plugin.

Authorizes are supported by the framework on its own, it'll be good to confirm if it works just in its capacity.

I'm getting the above permissions error.

What permissions error exactly?

[DrColza]

There are a couple of scenarios:

Given I am logged in as a read-only user
When I POST to the /update_user endpoint action
Then the update_user action is permitted

When I look in x-ray I can see that the customAuthorizer executes and correctly determines that the user is a read-only user.
In x-ray I can also see that the lambda executes and the user is updated in the database

Given I am authenticate using an invalid api-key
When I POST to the /update_user endpoint action
Then the customAuthorizer lambda fire and determines the user should be denied access
However the update_user action is permitted and executes

It's almost like both the function and customAuthorizer are being hit in parallel.

[medikoo]

@DrColza sorry, but it's not clear for me from above description, whether you're still facing any issues (?)

If you feel given fix is not sufficient can you explain why, and post an exact error you're receiving?

[DrColza]

The patch works for fixing the issue of being able to deploy. Approved.

With regards to the customAuthorizer error I'm getting, I will raise another issue if i need to. I need to spend more time troubleshooting it.

Thanks @medikoo