Mutliple API stages but in a single document - best practice

[mikebrules]

Hello - I read the Best Practice guidelines for SAM Online Tech Talk which states to create multiple environments in a single file, using, for example, Environment VARS passed in.

In practice, I'm not sure how this works. I clearly want to use Lambdas Versioning and Aliasing model, so I only want ONE defintion of my Lambdas, but I want MUTIPLE api stages.

If I run the template twice with a different ENV param, it will remove my previous stage, so do I create all my stages in the API definition (presumably using normal Cloudformation Type::Stage to reference the Serverless::Api)?

There seems to be lots of questions concerning versioning and environments. Any chance we could have a non-trivial example by an expert that could help clear this up please?

[zodeus]

I too would very much like to see some better examples. I've spent quite a lot of time just figuring out how all of these parts work together. I'm fairly certain now I have a reasonable grasp of the current situation. And there certainly seems to be a disconnect with what's being documented as capable and what actually makes sense in actual operations...

That being said @mikebrules, this is where I'm at. I have a script that wraps the cfn package/deploy which takes in any stage related variables i.e STAGE, VPCID, things like that.

Here it is

#!/usr/bin/env bash
set -e

if [ -z "$1" ]
  then
    echo "Argument stage is required"
    exit 1
fi

STAGE=$1
SERVICE="some-rando-name"
BUCKET="template-bucket"
REGION="us-west-2"

echo "CloudFormation packaging..."
aws cloudformation package \
    --region ${REGION} \
    --template-file sam.yml \
    --output-template-file ${STAGE}-packaged-template.yml \
    --s3-bucket ${BUCKET} \
    --s3-prefix sam/${SERVICE}

echo "CloudFormation deploying..."
aws cloudformation deploy  \
    --region ${REGION} \
    --template-file ${STAGE}-packaged-template.yml \
    --stack-name ${STAGE}-${SERVICE} \
    --capabilities CAPABILITY_NAMED_IAM \
    --parameter-override Stage=${STAGE}

I have a single SAM template that gets executed for each stage and creates a fully independent stack for each stage.

---
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Description
Parameters:
  Stage:
    Type: String
Resources:
  ApiGatewayApi:
    Type: AWS::Serverless::Api
    Properties:
      DefinitionUri: API.yml
      StageName: !Ref Stage
      Variables:
        Stage: !Ref Stage
        ThingGet: !Ref ThingGet
  FunctionRegistrantGet:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: ../../target/api.zip
      Handler: com.example.resource.Thing::get
      Runtime: java8
      FunctionName: !Join [ "", [ !Ref Stage, "-api-thing-get"]]
      Timeout: 60
      MemorySize: 256
      Role: !GetAtt DefaultRole.Arn
      Events:
        ProxyApiRoot:
          Type: Api
          Properties:
            RestApiId: !Ref ApiGatewayApi
            Path: /thing/{id}
            Method: GET
  DefaultRole:
    Type: "AWS::IAM::Role"
    Properties:
      ManagedPolicyArns:
          - "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole"
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          -
            Sid: "AllowLambdaServiceToAssumeRole"
            Effect: "Allow"
            Action:
              - "sts:AssumeRole"
            Principal:
              Service:
                - "lambda.amazonaws.com"
Outputs:
  ApiUrl:
    Description: URL of your API endpoint
    Value: !Join
      - ''
      - - https://
        - !Ref ApiGatewayApi
        - '.execute-api.'
        - !Ref 'AWS::Region'
        - '.amazonaws.com/'
        - !Ref Stage

And finally, here is the swagger doc used to generate the API Gateway API

swagger: '2.0'
info:
  title: Lasso API - ( STAGE )
  description: Description
  version: 1.0.0
schemes:
  - https
consumes:
  - application/json
produces:
  - application/json
basePath: /v1
paths:
  /thing/{id}:
    get:
      summary: Summary
      description: >-
        Description
      parameters:
        - in: path
          name: id
          type: string
          required: true
      x-amazon-apigateway-integration:
        type: aws_proxy
        uri: arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:ACCOUNT_ID_HERE:function:${stageVariables.ThingGet}/invocations
        httpMethod: POST
        responses:
          default:
            statusCode: 200
      responses:
        '200':
          description: >-
            Description

You may have noticed a few things here;

First off stages at this point in either Cloudformation native or SAM seems very clunky, remember cloudformation is toted as a tool for resource creation, not orchestration. You can feel it here...

Secondly, you'll notice I'm not using lambda versions as they were intended either. This one I think has a lot more technical issues that would need to be resolved to make sense entertaining. There is a serious breakdown in the pipeline of this, basically, you need

Lambdas -> Versions -> Aliases
API Gateway -> API Versions (???, think there is some concept here but it's unclear to me) -> Stages -> Lambda Aliases

all this seems fine and dandy in theory, all the parts are technically there... However, in practice, it breaks down in a number of spots which required special maneuvers (hacks?) to accomplish. All the way from requiring DSL's to produce CFN deployable templates to custom Lambdas used to transition/upgrade/roll forward your actual Lambdas (if you want versioning).

[mikebrules]

hi @zodeus thanks for the detailed post. Very useful indeed. I am in total agreement that the main breakdown is around managing versioning, specifically of Lambdas in my case which is causing me some confusion. Lambda out the box gives me a sensible versioning model with the use of aliases - I don't want multiple Lambda versions per environment, and nor, I expect, does AWS recommend that- its just that SAM is not yet able to support them.

I have managed to get Lambdas working with versions and aliases in SAM, but it is a mess in all honesty. I have a client who I want to ship a nice, tight set of cfn to, and at the moment, its hard to do that. I post my approach too as a contrast

[badfun]

@zodeus Does the template you show here actually work? I am doing something very similar, but I can neither get the local swagger file to work, nor get the stageName variables to show. For the local swagger file I get

'DefinitionUri' is not a valid S3 Uri of the form "s3://bucket/key" with optional versionId query parameter.

and for the stageVariable my lambda is named in the api as ${stageVariable.myFunction}. It doesn't actually get translated.

[badfun]

Never mind. I have worked past it.

[zodeus]

Ya, sorry about that, I had originally taken some production code and tried to obfuscate it. Clearly, I lost some details ;). Glad you worked through it.

[zodeus]

@badfun, in the AWS UI seeing that variable as the function name is correct, also the function trigger on the lambda will look broken, however, when a Stage is deployed, it populates that variable and everything gets linked together. Although through the UI it doesn't appear so.

[badfun]

Thanks @zodeus. I figured out that I was misunderstanding stage variables and it was actually working. For the api that I am creating I've gone with the 'Fn::Transform' 'AWS::Include' option. Seems to be working out so far.

[christianklotz]

After spending some time looking deeper into this, including coming across this issue, we wrote up a summary. Maybe you’ll find it interesting.

[mikebrules]

thanks @christianklotz I will take a good look

[rohitrishi]

Hello, I am new to CFN and AWS SAM.
I am also dealing with the need to deploy multiple stages which brought me to this post. But I had another slightly related question so I thought I would ask it here.
Is the "Events" property required for the lambda to be invoked via API gateway? All the examples that I have seen so far have required explicit declarations for the event type API for all the routes that are part of the swagger file.
I have about 14 paths/routes in my swagger file. Does that mean I have to do the following to enable invocation of the lambda from API Gateway:-

    Type: AWS::Serverless::Function
    .
    .
      Events:
        path1:
          Type: Api
          Properties:
            RestApiId: !Ref ApiGatewayApi
            Path: /foo
            Method: ANY
        path2:
          Type: Api
          Properties:
            RestApiId: !Ref ApiGatewayApi
            Path: /foo/bar
            Method: ANY
    ..... and so on for the remaining 12 paths

OR I can leave out the Events entry and putting in the apigateway-integrations in my swagger file will suffice for invoking the lambda for all the paths?

[mikebrules]

Hi @rohitrishi - I'm not sure this is what you wanted, but just in case - you can use the "lambda proxy" config to ensure every route on your api is directed to a single lambda function. I'm assuming you can mount this path like any other route and it should help trim down your function in cfn. Take a look at this

https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html#api-gateway-create-api-as-simple-proxy

Apologies if this isn;t what you wanted.

On the multiple api stages best practice, we (and quite a few others I know) just rolled over on that one. Its not easy by any means and the model is still confused. @christianklotz write up (^ up there) is worth a read for sure

[Jun711]

@mikebrules just wonder if u set your API gateway stage to use a specific version of your lambda.

I realized API Gateway always points to the latest version of my lambda so there is no point of manually publishing to a Prod stage after using CloudFormation to publish to a Dev stage. I am trying to figure out how to use lambda versioning properly with API Gateway stages

[mikebrules]

@Jun711 yes, you can do that. You just need to try and bind the Lambda references in your API Definition to stageVariables variables. Heres an example from a swagger file and a corresponding stack.yaml cfn that work together.

/articles:
  get:
    produces:
      - "application/json"
    parameters:
      - name: "from"
        in: "query"
        required: false
        type: "string"
     - name: "to"
       in: "query"
       required: false
       type: "string"
    responses:
    "200":
      ......
    x-amazon-apigateway-integration:
       responses:
          default:
          statusCode: "200"
          uri: "arn:aws:apigateway:eu-west-1:lambda:path/2015-03-31/functions/arn:aws:lambda:eu-west-1:020877680253:function:${stageVariables.ARTICLES_FUNCTION}:${stageVariables.ENV}/invocations"

Note the function:${stageVariables.ARTICLES_FUNCTION}:${stageVariables.ENV} value. When you set up the API in cfn, you add the stageVariables, like this;

 DevelopmentStage:
  Type: "AWS::ApiGateway::Stage"
  DependsOn: DevelopDeployment
  Properties:
    StageName: "Development"
    Description: "Development Stage"
    RestApiId:
      Ref: ApiGatewayApi
    DeploymentId: 
      Ref: DevelopDeployment
    Variables: 
      ENV: DEVELOP
      ARTICLES_FUNCTION: !Ref ArticlesFunctionName
      ARTICLES_ARTICLE_UUID_FUNCTION: !Ref ArticlesArticleUuidFunctionName
      ARTICLES_ARTICLE_UUID_RECALCULATE_FUNCTION: !Ref ArticlesArticleUuidRecalculateFunctionName
      AVERAGE_DI_FUNCTION: !Ref AverageDiFunctionName
      LOGIN_FUNCTION: !Ref LoginFunctionName
      PREDICTIVE_FUNCTION: !Ref PredictiveFunctionName

Does that make sense?

[Jun711]

@mikebrules Thanks Mike. It makes sense. 2 questions:
This is using lambda versioning / alias, right? Not the stages that we can see at API Gateway console?
and
when u r ready to deploy, you will publish a Production version of lambda and change the ENV variable to Production?

I am using SAM template and trying it with the following template:

myFunc:
    Type: AWS::Serverless::Function
    Properties:
      Handler: index.handler
      Runtime: python3.6
      MemorySize: 128
      FunctionName: functionName
      CodeUri: './'
      AutoPublishAlias: live
      Role:
        Fn::ImportValue:
          !Join ['-', [!Ref 'ProjectId', !Ref 'AWS::Region', 'LambdaTrustRole']]

and

 x-amazon-apigateway-integration:
                uri:
                  'Fn::Join':
                    - ''
                    - - 'arn:aws:apigateway:'
                      - Ref: 'AWS::Region'
                      - ':lambda:path/2015-03-31/functions/'
                      - 'Fn::GetAtt':
                          - myFunc
                          - Arn
                      - ':live/invocations'

[mikebrules]

HI @Jun711

I use API Gateway versioning alongside Lambda Versioning/ALiasing. My stack.yaml just includes multiple stage definitions. You can achieve what (I think) you want by defining a stage in your cloudformation that is constructed from environment variables - every time you run your deployment, the stack yaml is called with a new Environment version and the API Gateway stage (and Lambda mappings) are created just for that stage.

The other way is to define ALL stages in your stack, and use Conditions to only execute stage resource blocks based on the value of your ENV var...

[Jun711]

@mikebrules thanks. I will give it a try.
Btw, you mentioned the following, is it API Gateway StageName?
like

 Type: AWS::Serverless::Api
    Properties:
      StageName: Dev

If I run the template twice with a different ENV param, it will remove my previous stage, so do I create all my stages in the API definition (presumably using normal Cloudformation Type::Stage to reference the Serverless::Api)?

EDIT:
When I change stage name, it seems to delete my previous stageName.
I have Dev1 in AWS::Serverless::Api and if redeploy as Dev2, Dev1 stage will be deleted from Api Gateway.

[mikebrules]

HI @Jun711 yes, you've hit the part of the recommendation about defining everything once and passing in an ENV - it doesn't work like it should. Easiest solution is to do is define everything (all environments) in one stack.yaml file, but that means you've got ALL your api gateway envs in the file, but only a single version of the lambda - this is why it doesn;t hold together. Otherwise, what I now do is define an API PER environment (I don't use stages) and a Lamba per environment - it feels wrong, but its much, much easier....

[christianklotz]

@Jun711 you may be interested in our experience with multiple stages as described in this article.

The gist is we started with a single stack defining all environments, eventually moved away from it and have a single template that brings up one stack and can be repurposed for multiple environments. One of the main reasons is to limit the "blast radius" in case things go wrong during development – you don't accidentally want to bring down the production environment during a development deployment. Also this gives you much finer control over permissions, etc.

[Jun711]

Thanks @mikebrules @chrisradek I will check out your suggestions.

Btw, I noticed this. It could be a bug but if we manually deploy to another stage on API Gateway console, when you redeploy using a SAM yaml file, it doesn't seem to delete the stages.
On CloudFormation stack, it would say fail to remove this stage that you manually deployed. Have you guys noticed this?

[bluedusk]

Hi guys,
I'm new to CFN and SAM,

• One confusion I have is how to organize api.yaml files ? If I have 50 apis, do I put them in a single api.yaml or seprate them into 50 yaml and somehow combine them together ? From my understanding, one yaml file will generate one CFN template, is that correct ? Thanks.

[brettstack]

This is an important feature and I'll make sure we get it scheduled. In the meantime, I'd appreciate any suggestions/ideas people have of what the schema/API should look like. Perhaps a Stages property which is an array of objects which will allow for more control over a stage's properties?

[bluedusk]

Current if I create two API with different stageName, it will create two RestApi with these stages, not two stages in one RestApi. Seems no way to have two stages in one Api.

[neeleshs]

@badfun , you said you've worked past the issue of stageVariable not resolving in swagger yaml. It would be great help if you can comment on how you did that. I'm facing the exact same issue with function name resolving to ${stageVariable.Stage}_<function_name> instead of resolving stageVariable.Stage.
Thanks!

[kaitea]

Hi, is there an ETA for this fix? This really impedes development and adopting SAM and leveraging infrastructure as code. Thanks.

[gpasq]

What a ridiculous mess to just do dev/test/stage/production.

[metaskills]

I'm not sure what the mess is or what AWS or SAM should do. I'm having a great time using the single stack arch in this article (https://hackernoon.com/managing-multi-environment-serverless-architecture-using-aws-an-investigation-6cd6501d261e) with a StageName parameter that helps drive everything from tagging to other parts of my template using intrinsic functions.

[brettstack]

@gpasq for multi-stage/environment development it's best to spin up a new stack for each environment/stage, and if you have the means to, they should also be deployed into separate AWS accounts for more isolation. We certainly need to improve our documentation around this (and more).

[Tanbouz]

How do you manage environment variables?

Assuming a single template + multiple stage deployments is the recommended way with a STAGE parameter that can be either dev, staging or prod.

Maintain 10+ or 20+ variable long --parameter-overrides for 3 different stages? 🤕

Thinking of just using STAGE in the template, integrate SSM into application code and then load SSM configs depending on the value of STAGE.

[mikebrules]

@Tanbouz there is the option of using Conditions to manage this kind of stuff, but in all honesty, its going to be a nightmare. I gave up a year ago on managing multiple API Gateway stages. The model just isn't there and the hacks are awkward for little actual benefit. We deploy an API per environment, and a Lambda per Environment, which feels broken from a 12-factor app model, but is actually very workable, and you won't regret it once you've got over the guilt :)

[christianklotz]

@Tanbouz it can be a bit misleading that API Gateway and Lambda both have their own concepts of environments (stages and aliases) but it may be better not to follow them and go for the one-stack-per-environment approach (see my previous comment).

There are still good use cases for the use of stages and aliases – AWS just gives you plenty of options which can be a bit overwhelming at times.

[gpasq]

I’m using Serverless now, which is one stack per environment, and it works very well. Trying to use multiple environments in a single stack is infeasible, IMO Regards -Greg Pasquariello

…

On Oct 2, 2018, at 2:47 PM, Christian Klotz ***@***.***> wrote: @Tanbouz <https://github.com/Tanbouz> it can be a bit misleading that API Gateway and Lambda both have their own concepts of environments (stages and aliases) but it may be better not to follow them and go for the one-stack-per-environment approach (see my previous comment). There are still good use cases for the use of stages and aliases – AWS just gives you plenty of options which can be a bit overwhelming at times. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#198 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADoBfQU5PCKDhLSsk8ovn-8UNBB9SxZmks5ug9DRgaJpZM4PXWo8>.

[Tanbouz]

multiple stage deployments

Sorry, I was not clear. I meant multiple environments or stacks.

I already started refactoring my template as everyone here suggested. Till now it is looking great 👍 . The discussion here and resources linked were extremely helpful, thanks.

---

I was wondering how is everyone managing their SAM serverless application configs when using this pattern.

A large number of Lambda environments variables used as configs for a serverless application are a hassle to manage. I need to pass different values per environment into a single template that takes the parameters over the command line.

Prefer to

• avoid putting application configs in long deployment commands.
• version control configs.
• use a consistent config format across development and deployment.
• have config isolation between environments.
• unify and share common configs across the application.

Options I'm currently aware of...
** StageName here refers to the suggested concept of isolated environments or "stacks" and not API gateway stages.

(1) Include configs in application code. Load appropriate configs per StageName

• Difficult to share configs across Lambda functions unless each Lambda function includes the whole application code. Goes against best practice
• Security risk (all code paths have access to all configs).

(2) --parameters-override

• SAM local support Support !Ref of template parameters aws-sam-cli#657 matching sam deploy format.
• Possible JSON support Support in parameter files in --parameter-overrides aws-cli#3445
• Until start-api and deploy get parameter file support, you have to manage parameter values over command line for different environments or develop custom scripts to do so.

(3) CloudFormation dynamic SSM references

• Looks like a new feature, not sure if supported by SAM.
• Or if I can plug in StageName using !Sub into the reference.
• Limited to 60 references.

(4) Get configs using SSM API during runtime.

• Only need Lambda environment variables to store StageName.
• Separates application level variables from resource level variables.
• Extra dependency on SSM.
• Requires implementation in application code.
• 70ms-100ms cold start latency hit.
• Can use IAM policy to restrict access to an SSM path prefixed with StageName.
• Requires connectivity and access to SSM service (Local SSM Parameter Store aws-sam-cli#616).

(5) --parameters create-change-set

• Pass a different JSON parameter files for each environment.
• You have to use create-change-set and execute-change-set instead of aws or sam's deploy
• Does not work with sam local

(6) CodePipeline's Template configuration

• Build a pipeline to handle (5) automatically.
• JSON parameters can be included with the application code or stored separately to bypass the con stated here

(7) --env-vars

• Only works with SAM local.
• Defined per function, have to mix and match between JSON and shell global variables. Kind of a mess...

(8) Template parameter defaults

• Can only be used once

(9) CloudFormation conditions

• I agree with @mikebrules, it will be a mess.

---

• Currently leaning towards (8) for development using sam local since Support !Ref of template parameters aws-sam-cli#657

• (6) for anything that needs to be deployed on AWS as suggested by SAM online tech talk. Then I can avoid --env-vars, --parameters-override, complex deployment commands and including configs within application code. Simply override the defaults using a JSON file that I could easily manage and update for different environments.

• (2) looks promising if JSON support is added.

I will give it a shot in the coming days and see how it goes.

It was not as clear until I wrote this up. I felt overwhelmed as with API stages and Lambda aliases. Any other recommendations/suggestions are welcome.

[Joycechocho]

@Jun711 were you still using the single stack for multi-stage api gateway deployment? I was running into the same issue where the SAM template remove my previous stage on Api gateway if I deploy it on a second time with a different StageName. Any workaround will be appreciated!

[lucky2siddharth]

@Jun711 were you still using the single stack for multi-stage api gateway deployment? I was running into the same issue where the SAM template remove my previous stage on Api gateway if I deploy it on a second time with a different StageName. Any workaround will be appreciated!

I'm too facing the same issue. SAM template deployment replaces my previously created stage with new one.

[Jun711]

@Joycechocho @lucky2siddharth
Right, it would remove your previously deployed stage if the stage name is changed.
If you want to deploy to other stages, you can do it manually on API Gateway console.

However, I have learnt from others from this discussion and started deploying a Lambda and an API for each environment(one stack per environment). It is easier to manage.

You can check out Continuous Integration and Continuous Deployment(CI/CD) using AWS CodePipeline and AWS CodeStar to see how to set up.

[AvinashDalvi89]

any one got any solution to handle multiple stages under API gateway using serverless SAM.

[patrickrapid]

nt. Also this gives you much finer control over permissions, etc.

I agree with the approach of :
Don't deploy a SAM API with stages, instead deploy multiple copies of the same API and associated resources (lambda functions).

In my environment I have 3 separate pipelines, each connected to a different branch of my source. I pass the parameter at codebuild time. This works well, and doesn't mangle all three environments in case of a "stack rollback".

My codepipeline and codebuilder are also templated to accomplish this feat with a single parameter "dev","prod",or "qa".

[jfuss]

Looks like the initial ask/question has been answered in this thread, namely SAM's current design only accepts a single Stage and multiple stages should be another stack. I realize this isn't everyone's setup but is the current approach we have taken. If there is an ask to support multiple stages, please create a new issue (there might be another issue already for this, but haven't checked).

Closing as the initial issue was answered