SAM deploy doesn't set environment variables

[chrisfosterelli]

Description

When using AWS SAM for local development, I can introduce environment variables by setting them in the template with no value, and then defining them in my environment. However, when I go to deploy, the environment variables do not appear to be inserted by sam package or sam deploy, and I get the following error on deploy:

Failed to create the changeset: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state Status: FAILED. Reason: [/Resources/GeneratePlan/Type/Environment/Variables/SECRET_ACCESS_KEY] 'null' values are not allowed in templates

Where SECRET_ACCESS_KEY is one of my environment variables. I cannot find any documentation detailing how to deploy projects with environment variables, either by having them defined in my environment or providing them in an alternate config.

I don't want to add the environment variables to my template.yml directly because this is stored in Git, and I don't want to edit them manually into the packaged.yml file each time between the package and deploy steps as that's cumbersome.

I haven't seen any steps in the documentation or similar issues, so I presume this is either an edge case bug or I am just missing something simple (in which case I might file this as a documentation bug) 😄

Steps to reproduce

The following config is a minimal excerpt from mine:

AWSTemplateFormatVersion: '2010-09-09'

Transform: AWS::Serverless-2016-10-31

Globals:

  Function:
    Environment:
      Variables:
        SECRET_ACCESS_KEY:

Resources:

  AnswerChallenge:
    Type: "AWS::Serverless::Function"
    Properties:
      Runtime: nodejs8.10
      Handler: build/functions/answerChallenge.default
      CodeUri: ./
      Policies: AmazonDynamoDBFullAccess
      Events:
        GetRequest:
          Type: Api
          Properties:
            Path: /event
            Method: get

Observed result

When I run sam start-api with SECRET_ACCESS_KEY defined, the lambda works as expected. When I attempt to deploy with sam package and sam deploy, I receive an error about undefined variables.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: MacOS
2. sam --version: 0.11

[jsonmaur]

Same issue here. sam package/sam deploy should accept the --env-vars flag.

My current workaround is to use CF parameters, since sam package/sam deploy are aliases for cloudformation package/cloudformation deploy.

template.yml

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  SomeFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: some-function/
      Handler: index.handler
      Runtime: nodejs8.10
      Environment:
        Variables:
          SOME_VAR: !Ref SomeVar
Parameters:
  SomeVar:
    Type: String

$ sam package --template-file template.yml --s3-bucket BUCKET_NAME --output-template-file packaged.yml

$ sam deploy --template-file packaged.yml --stack-name STACK_NAME --capabilities CAPABILITY_IAM --parameter-overrides SomeVar=123

[chrisfosterelli]

@jsonmaur Thanks for sharing a workaround! Do you know if that is compatible with local development (i.e. start-api and start-lambda)?

[jsonmaur]

Works fine for me when I run sam local invoke --env-vars env.json. --env-vars seems to override the parameters.

[Marclev78]

Is there any response to this? I'm seeing the same thing.

sam package creates a YAML file containing my environment variables as expected, but after running sam deploy I can see that the Lambda configuration in the AWS console hasn't been updated to include them.

Trying to execute the Lambda then fails with the predictable "environment variable not defined" result.

[ivanmarques]

Hi!

Any updates on this issue?

Thanks!

[whereisaaron]

Wow does SAM make setting environment variables complicated!

@jsonmaur's workaround seems the best, using the (undocumented in sam docs) --parameter-overrides argument to sam deploy using the Name1=Value1 Name2=Value2 syntax.

sam build documents a --parameter-overrides argument, but it uses the the JSON shorthand syntax: ParameterKey=KeyPairName, ParameterValue=MyKey ParameterKey=InstanceType, ParameterValue=t1.micro .

So sam and cloudformation have multiple same-named arguments with entirely incompatible value syntax? Not helpful!

I tried using the sam build --parameter-overrides option but the parameters don't see to go into the build, and the values you specify for build are lost when you deploy. So not even sure what that is for?

[chrisfosterelli]

Yeah the sam local start-api also uses the mismatched format in --parameter-overrides, which is frustrating! Having to use two different config formats between local dev and prod deploy is not great... I had tried the --env-vars solution but had issues with the different cases expected between uppercase and snake case. This is what I had to do to get some resemblance of consistency:

I store my config file as a JSON file, such as config.json:

{
  "Region": "us-west-1",
  "Secret": "my-real-secret",
  [...]
}

And define them as parameters in the template.yaml:

Parameters:
  Secret:
    Type: String
  [...]

[... later ...]

Globals:
  Function:
    Environment:
      Variables:
        SECRET: !Ref Secret
        [...]

Then for development I have the following in package.json:

"scripts": {
  "api": "sam local start-api --region us-west-2 --parameter-overrides \"$(jq -r -j 'to_entries[] | \"ParameterKey=\\(.key),ParameterValue=\\(.value) \"' config.json)\"",
}

So I can run npm run api to get the api server going with the config file.

Then for deployment our CI runs:

> sam package --template-file ./template.yaml --s3-bucket bucketname --output-template-file ./packaged.yaml
> sam deploy --template-file ./packaged.yaml --stack-name stackname --capabilities CAPABILITY_IAM --parameter-overrides $(jq -r 'to_entries[] | "\(.key)=\(.value)"' config.json)

Surprising amount of work to get sane environment variables in SAM... I think this needs some developer experience TLC or maybe I'm missing something blatantly obvious 😅

[L-F-Escobar]

this is an issue. My Environment.variables just disappear upon command sam deploy --template-file packaged.yaml --region us-west-2 --capabilities CAPABILITY_IAM --stack-name lambda-stack-test

  LambdaStackFunction:
Type: AWS::Serverless::Function
Properties:
  FunctionName: LambdaStack-Local
  Timeout: 15
  Description: Local stack use to test -> deployed version
  CodeUri: lambda_stack/
  Handler: main.handler
  Role: !GetAtt LambdaExecutionRole.Arn
  Runtime: python3.7
  Layers:
    - arn:aws:lambda:us-west-xxxxxxxxxxxxxxx
    - arn:aws:lambda:us-west-xxxxxxxxxxxxxxx
Environment:
  variables:
    S3_ENDPOINT: https://s3.us-west-2.amazonaws.com
    SQS_ENDPOINT: https://sqs.us-west-2.amazonaws.com
    REGION: us-west-2
    TIMEZONE: US/Pacific

[whereisaaron]

@LEscobar-Driver capital ‘V’ in Variables

[L-F-Escobar]

@whereisaaron does nothing at all. Still the same

[awood45]

For local testing, you can override env variables in a few ways, including the --env-vars parameter. However, in the deploy step, you're just passing along the packaged template. For that, you must define all environment variables, either directly or as a parameter.

If this doesn't cover your needs, it could help to explain more about what you're trying to do.

[chrisfosterelli]

@awood45 Thanks for jumping in! I will see if I can provide some context. Essentially, I'm looking for a unified method for configuring the application that works both in sam local and sam deploy.

For example, when I create a non-serverless Node.js application based on express, in the documentation I can provide information on what environment variables are expected to be present. Developers can define these in their environment for local dev, and in production the server can set those environment variables. It's the same interface, it's well defined intuitively, and it's conventional.

I thought I had similar behaviour in SAM, as I found a Github issue mentioning how to pass through environment variables by setting them to an empty value in the template. That was great! It was frustrating to learn this does not work when deploying.

Some notes on my goals:

• I do not want to put the environment variables directly into the configuration file, as I can then no longer safely check that file into version control
• There are a lot of environment variables in any moderately complex app. I'd like to not have to supply all of them on the command line, or use a hack like jq to get them to parse from a file. This is particularly annoying as sam local and sam deploy expect them on the command line in different formats.
• I'd like to be able to use the same interface / format for describing my environment variables in both local dev and when deploying. Setting environment variables is the ideal method here for me personally but even a common format config file would be significantly better.

FWIW I think there is both dev UX aspects and documentation aspects here. I have ideals about how to deploy, which would be great if we could improve the dev UX there, but realistically documentation is what originally made me create the issue here. When I went to figure out how to add environment variables in dev I couldn't find any documentation on this at all. I did find an answer through Github issues. When I found that didn't work in production, I again couldn't find any documentation at all on how to actually deploy environment variables, and had to get help on this issue here from @jsonmaur to get deploy working. It may seem intuitive as it uses another AWS command under the hood, but do consider that users of SAM for the first time might not be familiar with how to deploy lambda's under cloudformation. This has been my experience over many months so it's possible that docs have improved since then but I definitely found it unclear how to actually deploy an app with env variables.

Does that clarify a bit? Happy to help elaborate some more if needed.

[whereisaaron]

I too expected SAM would have a consistent way to handle env vars between running locally and deploying. Like @jsonmaur I expected --env-vars to work for deployment too.

Right now sam local, sam build, and sam deploy have three, mutually incompatible ways to specify environment variable values.

[awood45]

We'll definitely want to improve documentation on the deploy difference, though I would agree we should at minimum converge options on build/local to match.

One issue with deployments is that they are simply a CloudFormation deployment. What you'd be asking for in that case is for --env-vars to transform the template before deployment, without those changes being saved anywhere.

Unfortunately the ship has sailed to an extent for naming this better without it being a breaking change, but it's really a way to override your production env variables for local testing.

[jfuss]

To add more context to the "We have 3 different ways to do environment variables".

The --env-vars options pre-dates use resolving Parameters. During that time, we needed a way to allow customers to change their Env Vars defined in the template. This still has applications today (overriding dynamic references that are inlined in the Env Var definition for example) but most of the time --parameter-overrides will be what you will want. Especially, if you are following the pattern of using Parameters to allow changes to the template at deploy (or in this case local invoke) time.

Now to the "build and local have different way than deploy for parameter-overrides.

Sadly, we didn't have a better way. Currently deploy still shells out to the AWS CLI (which we will be addressing separately). When we introduced the --parameter-overrides into the local suite (and later build), we attempted to match what AWS CLI was doing to keep things consistent. AWS CLI, however, does not use click (they use argparse if I recall correctly). Because the the different ways we have the cli command defined, we couldn't match AWS CLI's way and were forced to deviate. When we revisit deploy, this will most likely change to follow the pattern we have in local/build.

Hopefully that sheds some light on why, it is what it is right now. We know the inconsistency is a UX problem and rough edge right now.

[chrisfosterelli]

Thank you both for the context!

[whereisaaron]

Thanks for the background. Everyone here seems to agree (!) that --parameter-overrides on sam deploy to set environment variables in the template via parameters is the current best way to go. However that option is not listed in the command or online documentation:

https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-deploy.html

[adnangul]

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  SomeFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: some-function/
      Handler: index.handler
      Runtime: nodejs8.10
      Environment:
        Variables:
          SOME_VAR: !Ref SomeVar
Parameters:
  SomeVar:
    Type: String

I'm trying to follow the example you shared above, but instead of the value of the parameter, the name of the parameter itself getting into the env variable i.e., SOME_VAR = 'SomeVar', also I want to set the default value of parameters; based on my understanding it should be done as below but none of the way it is working

**Parameters:

SomeVar: value**

[wojtekk]

I agree that documentation could be better - the most important is to add missing param --parameter-overrides to aws deploy command.

sam deploy is an alias to aws cloudformation deploy, I think - when you run aws cloudformation deploy help you can find information about missing param.

Working example

template.yaml

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
  
Globals:

  Function:
    Timeout: 3

Parameters:

  SomeVar:
    Type: String
    Description: My SomeVar
    Default: default value

Resources:

  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: hello-world/
      Handler: app.lambdaHandler
      Runtime: nodejs12.x
      Environment:
        Variables:
          SOME_VAR: !Ref SomeVar
      Events:
        HelloWorld:
          Type: Api
          Properties:
            Path: /hello
            Method: get

Outputs:

  HelloWorldApi:
    Description: "API Gateway endpoint URL for Prod stage for Hello World function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"

  HelloWorldFunction:
    Description: "Hello World Lambda Function ARN"
    Value: !GetAtt HelloWorldFunction.Arn

  HelloWorldFunctionIamRole:
    Description: "Implicit IAM Role created for Hello World function"
    Value: !GetAtt HelloWorldFunctionRole.Arn

When you run:

sam build
sam local invoke HelloWorldFunction -e events/event.json

the default value will be used - in our case: SOME_VAR=default value

To deploy run commands:

sam package --template-file template.yaml \
  --s3-bucket my-bucket \
  --output-template-file packaged.yaml

sam deploy --template-file packaged.yaml \
  --stack-name my-application \
  --capabilities CAPABILITY_IAM \
  --parameter-overrides SomeVar=other_value

The last line do "the magic" and allow you to override default value:

--parameter-overrides SomeVar=other_value

When the real function will be invoked: SOME_VAR=other_value.

I think is important to remember that SAM templates are compatible with CloudFormation templates and all techniques/concepts could be used in SAM development.

[peterkulik]

Thanks @wojtekk! It's really helpful!

[mvn-hoangnguyen-hn]

I get "unable to use container host config override file from '$HOME/.config/aws-sam-local/container-config.json': HOME env variable is not set" after get request "127.0.0.1:3000/ping" when use "sam local start-api --template sam.yaml".

This is my log:
"2020/02/28 10:22:06 Invoking com.amazonaws.serverless.proxy.struts2.Struts2LambdaHandler::handleRequest (java8)
2020/02/28 10:22:07 WARNING: No AWS credentials found. Missing credentials may lead to slow startup times as detailed in #134
2020/02/28 10:22:07 Mounting /c/Users/huyho/OneDrive/Documents/Java Project/aws-serverless-struts2-archetype as /var/task:ro inside runtime container
2020/02/28 10:22:07 unable to use container host config override file from '$HOME/.config/aws-sam-local/container-config.json': HOME env variable is not set
2020/02/28 10:22:07 Error invoking java8 runtime: Error: No such image: lambci/lambda:java8 "

I have followed the guide in README.md. Please help me.

[rebeccapeltz]

Local env variables work fine with sam local invoke -n env.json where
env.json looks like this

{
  "Parameters": {
    "KEY1: "value1"
    "KEY2": "value2"
    ...
  }
}

I can't figure out how to list the K/V pair the parameter-overrides. I've tried many permutations. Here's one

sam deploy \
  --template-file package.yml \
  --stack-name my-stack \
  --s3-bucket my_bucket \
  --region us-east-1 \
  --capabilities CAPABILITY_IAM \
  --parameter-overrides KEY1=valu1e, KEY2=value2, KEY3=value3

I've setup env variables in the web UI lambda definition area but that doesn't seem to help.
The --help is not very helpful for this parameter. I don't know what ParameterKey=KeyPairName, ParamterValue=MyKey means.
What I see in. help for deploy:

 --parameter-overrides           Optional. A string that contains AWS
                                  CloudFormation parameter overrides encoded
                                  as key=value pairs.For example, 'ParameterKe
                                  y=KeyPairName,ParameterValue=MyKey Parameter
                                  Key=InstanceType,ParameterValue=t1.micro' or
                                  KeyPairName=MyKey InstanceType=t1.micro

Can I get some help with how to list multiples K/V pairs in the parameter-overrides

[adnangul]

I think one fo the option is, you can set the env variables and then use the variable while executing the command. Or create a shell script wrapper which can supply these value from env. Ideally once you are done with development, these variable will be injected through ci/cd pipelines Regards Adnan

…

On Tue, Aug 4, 2020 at 11:27 AM Nick Bordeau ***@***.***> wrote: Has anyone found a good way to address this with quite a few different env variables (say 10-20 or so)? The two things I want to avoid are: - I definitely do not want to commit any files containing my env variables to a repo. - I do not want to manually type them on every deploy. That seems like a living nightmare. The only ways I could think of would be adding them to the template.yaml file or adding the script with the --parameter-overrides to package.json file. Both would mean I have to commit my env variables to a repo. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1163 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACIH7AHZUTZWG5WEFCCIEYTR7AZITANCNFSM4HK5LDNA> .

-- Regards, *Adnan Yaqoob*

[jeremydrichardson]

After reading through this thread, I think one main problem is the docs for --parameter-overrides are extremely confusing.

'ParameterKey=KeyPairName, ParameterValue=MyKey ParameterKey=InstanceType,ParameterValue=t1.micro'.

Whereas it should be simply:

ParameterKey1=Value1, ParameterKey2=Value2

[m-sureshraj]

@Borduhh

Committing any files containing my env variables to a repo.

Don't commit your env file, instead maintain a sample env file for others to use.

# .gitignore
env.json

For more info, refer to the documentation.

// env.json
{
  "YourFunctionName": {
    "GITHUB_CLIENT_ID": "foo",
    "GITHUB_CLIENT_SECRET": "bar"
  }
}

Others can create their env file based on the sample file.

// env.sample.json
{
  "YourFunctionName": {
    "GITHUB_CLIENT_ID": "",
    "GITHUB_CLIENT_SECRET": ""
  }
}

Manually typing env variables on every deploy. That seems like a living nightmare.

Indeed. But you can maintain a deployment script for this. Something like this would work.

// package.json
"scripts": {
    "start": "sam local start-api -t ../template.yaml -n ../env.json",
    "local-invoke": "sam local invoke -t ../template.yaml -n ../env.json functionName",
    "deploy": "path/to/deployment-script.sh"
}

Hope this helps.

[bvisan]

@m-sureshraj

I've got the same setup you just mentioned, but still no luck. Variables are showing as undefined on local-invoke.

Is there something I am missing? Does my template file need something added as well?

Thank you!

[m-sureshraj]

Hi @bvisan,

The local-invoke command works fine on my local. Did you list your environment variables name in the template?
Here is my template.yaml

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Obtain access token from Github

Globals:
  Function:
    Timeout: 30 # Seconds

Parameters:
  GITHUB_CLIENT_ID:
    Type: String
    Description: Github client id
  GITHUB_CLIENT_SECRET:
    Type: String
    Description: Github client secret

Resources:
  getAccessTokenFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: github-oauth/
      Handler: app.getAccessToken
      Runtime: nodejs12.x
      Environment:
        Variables:
          GITHUB_CLIENT_ID: !Ref GITHUB_CLIENT_ID
          GITHUB_CLIENT_SECRET: !Ref GITHUB_CLIENT_SECRET
      Events:
        getAccessToken:
          Type: Api
          Properties:
            Path: /access_token
            Method: get

[bvisan]

@m-sureshraj That did the trick 👍🏼

[ozbillwang]

So why we can't set Parameter overrides directly in samconfig.toml file as well?

--parameter-overrides is inconvience, because I need set 20+ parameter key pair.

and these key/values are plain text, no sensitive password, So I prefer to put in samconfig.toml directly, such as

[dev]
[dev.deploy]
[dev.deploy.parameters]
stack_name = "hello_world"
s3_prefix = "hello_world"
region = "us-east-2"
confirm_changeset = true
capabilities = "CAPABILITY_IAM"
parameter1 = "value1"
parameter2 = "value2"
parameter3 = "value3"
parameter4 = "value4"
parameter5 = "value5"
parameter6 = "value6"
parameter7 = "value7"
parameter8 = "value8"
parameter9 = "value9"
parameter10 = "value10"
parameter11 = "value11"
...
[staging]
...
[prod]
...

and I can easily add other environments as staging, prod, etc.

Update 1

I made it. We can set parameter_overrides directly in samconfig.toml

$ cat samconfig.toml

[dev]
[dev.deploy]
[dev.deploy.parameters]
stack_name = "hello_world"
s3_prefix = "hello_world"
region = "us-east-2"
confirm_changeset = true
capabilities = "CAPABILITY_IAM"
parameter_overrides = "ParameterKey=parameter1,ParameterValue=value1 ParameterKey=parameter2,ParameterValue=value2"
...
[staging]
...
[prod]
...

When sam deploy --config-env dev, I can see the part of Parameter overrides is not empty any more.

Of course, you can add --parameter_overrides in command line as well for some of them.

For the parameter_overrides entry, both the parameter values that you provide on the command line and entries in the configuration file take precedence over corresponding objects declared in the Parameters section of the template file.

reference: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html

Update 2

I found an easy way to set parameter_overrides in samconfig.toml (Notice the escape)

parameter_overrides = "parameter1=\"value1\" parameter2=\"value2\""

Both ways are working

Update 3

I made a repo to show case on how to manage the parameter with seperate files for different environments.

Take a look, if you are interested.

https://github.com/ozbillwang/sam-hello-world

[jeremydrichardson]

Right. I think the main reason against using the samconfig.toml is the secrets issue since you will most likely want to commit it to git. If you add to .gitignore than no problems, works great. Otherwise you could have secrets committed to your repo.

Of course creating scripts with the parameter overrides in your package.json will also add secrets to your git repo as well.

Dotenv and cross-var

I prefer using a .env file then using doting and cross-var to create scripts in my package json.

yarn dotenv -e .env.prod -- cross-var sam deploy --stack-name NAME-OF-STACK --parameter-overrides NodeEnv=prod parameter1=%PARAMETER1% parameter2=%PARAMETER2% parameter3=%PARAMETER3%

Then you can have multiple .env files for each environment (.env.dev, .env.prod, .env.uat) and create a package json script for each (I use start for dev and sam local start in the script instead of sam deploy, deploy-prod, deploy-uat).

That way I am using an industry standard .env file that can hold secrets and even though the package.json scripts can get quite large, I'm not editing them often so I can just use the yarn scriptname and I'm good to go.

Then it's always good practice to add .env.example files as well that will be committed to git so anyone cloning the repo will at least know what environment variables are required.

Hope that gives another possible way of organizing.

[classforma]

How do you update the MongoURI parameter value in sam build?

Parameters:
  MongoURI:
    Type: String
    Description: MongoDB Connection String
Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      Environment:
        Variables:
          mongo_uri: !Ref MongoURI

We are using CodePipeline for deploying and using sam deploy is not option.

I have tried sam build --parameter-overrides 'ParameterValue=MongoURI,ParameterValue=test_value' but it is not helping.

[jeremydrichardson]

I believe the format of your parameter overrides is incorrect. The SAM documentation is super unclear on the syntax and I think it's actually wrong. If you look at my example above parameter overrides should just be key value pairs with no commas or quotes.

For your example I would try:

sam build --parameter-overrides MongoURI=mongouristring

[classforma]

I believe the format of your parameter overrides is incorrect. The SAM documentation is super unclear on the syntax and I think it's actually wrong. If you look at my example above parameter overrides should just be key value pairs with no commas or quotes.

For your example I would try:

sam build --parameter-overrides MongoURI=mongouristring

Does not work either unfortunately.

[jfuss]

Quick clarification here and likely we need to update some docs.

--parameter-overrides does not set the values in the template on build or deploy. On sam build, parameter-overrides is used as a way to tell SAM CLI to resolve the parameters in the template so the build does not fail (like passing is a parameter for CodeUri or Runtime, etc. On sam deploy, parameter-overrides is passed to CloudFormation who does the full resolving.

parameter-overrides is not use to set the value back into the template.

[Borduhh]

Quick clarification here and likely we need to update some docs.
--parameter-overrides does not set the values in the template on build or deploy. On sam build, parameter-overrides is used as a way to tell SAM CLI to resolve the parameters in the template so the build does not fail (like passing is a parameter for CodeUri or Runtime, etc. On sam deploy, parameter-overrides is passed to CloudFormation who does the full resolving.
parameter-overrides is not use to set the value back into the template.

Thanks for the clarification. We are using CodePipeline for deploying which is not compatible with sam deploy.

Is there any way to set these parameters dynamically with sam build or sam package? We would like to ingest environment secrets as environment variables, this is very straightforward with docker but we can't figure it out with sam

The way we ended up doing it was using Parameter Store to store all variables and then you can update based on the stage. We migrated over to Serverless now because of a different use case but you should be able to do the same with SAM. Here is what it looks like:

custom:
  stage: ${file(../../serverless.common.yml):custom.stage}
  ###################
  # These variables are stored in a common YAML file but I've added the actual values here. Not sure if that works with SAM, but should since it's just YAML.
  resourceStageList:
    production: production
    sandbox: sandbox
    dev: dev
  resourceStage: ${self:custom.resourceStageList.${self:custom.stage}, self:custom.resourceStageList.dev}
  ###################


  mongodbConnectionUrl:
    dev: ${ssm:/appName/serviceName/dev/mongo-connection-url~true}

  environment:
    MONGO_CONNECTION_URL: ${self:custom.mongodbConnectionUrl.${self:custom.resourceStage}}

[jfuss]

@classforma You can achieve this by doing CodePipeline Stage -> CodeBuild -> sam deploy --parameter-overrides. You can also drop into aws cloudformation deploy is that is what you really want to do. These are the typical options and how CloudFormation supports what you are trying to achieve. SAM will not produce a template with replaced Parameters. SAM is built on CloudFormation and therefore we follow CloudFormation mechanisms.

[mudassirkhan19]

For some reason having underscores in parameter values doesn't let me set parameter-overrides for that parameter, changing it to CamelCase did the trick for me. Can anyone confirm if that's the case?

Hi @bvisan,

The local-invoke command works fine on my local. Did you list your environment variables name in the template?
Here is my template.yaml

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Obtain access token from Github

Globals:
  Function:
    Timeout: 30 # Seconds

Parameters:
  GITHUB_CLIENT_ID:
    Type: String
    Description: Github client id
  GITHUB_CLIENT_SECRET:
    Type: String
    Description: Github client secret

Resources:
  getAccessTokenFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: github-oauth/
      Handler: app.getAccessToken
      Runtime: nodejs12.x
      Environment:
        Variables:
          GITHUB_CLIENT_ID: !Ref GITHUB_CLIENT_ID
          GITHUB_CLIENT_SECRET: !Ref GITHUB_CLIENT_SECRET
      Events:
        getAccessToken:
          Type: Api
          Properties:
            Path: /access_token
            Method: get

[matt-dalton]

Coming to this issue from Google, I'm still confused what the intended way to handle env vars is. I tried something like this (based on a few blogs):

Parameters:
  DEBUG:
    Description: 'Required. In debug mode will log more verbosely etc.'
    Type: 'Boolean'
  API_KEY:
    Description: 'Required. The key for the API.'
    Type: 'String'

Globals:
  Function:
    Environment:
      Variables:
        DEBUG: !Ref DEBUG
        API_KEY: !Ref API_KEY

I then want to be able to configure a different config (i.e. a set of env vars) for:

• local development using sam build and sam local invoke
• Running unit tests
• Staging (deployed via SAM to ECS)
• Prod

I would imagine this is an MVP setup for any project.

We're also using Python if that makes a difference.

Is there a simple way to do this? I thought the above would work, but when using build + invoke the environment variables are set to their names (e.g. os.environ['API_KEY']='API_KEY').

btw @mudassirkhan19, it didn't work for me even with CamelCase.

[bvisan]

For some reason having underscores in parameter values doesn't let me set parameter-overrides for that parameter, changing it to CamelCase did the trick for me. Can anyone confirm if that's the case?

Hi @bvisan,
The local-invoke command works fine on my local. Did you list your environment variables name in the template?
Here is my template.yaml

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Obtain access token from Github

Globals:
  Function:
    Timeout: 30 # Seconds

Parameters:
  GITHUB_CLIENT_ID:
    Type: String
    Description: Github client id
  GITHUB_CLIENT_SECRET:
    Type: String
    Description: Github client secret

Resources:
  getAccessTokenFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: github-oauth/
      Handler: app.getAccessToken
      Runtime: nodejs12.x
      Environment:
        Variables:
          GITHUB_CLIENT_ID: !Ref GITHUB_CLIENT_ID
          GITHUB_CLIENT_SECRET: !Ref GITHUB_CLIENT_SECRET
      Events:
        getAccessToken:
          Type: Api
          Properties:
            Path: /access_token
            Method: get

I think you are right. I'm revisiting this with the parameter-overrides in mind now and using '_' in the variable names template does not reference properly :/

From this post, seems that the underscore is considered a non-alphanumeric value.

Looks like only whats on the right side of the declaration matters, the Variable name in the template can have them.

[jfuss]

Reading through the comments, I believe this is solve.

To summarize: SAM CLI will not set environment variables in the template. Instead for sam deploy you use --parameter-overrides, which tells CloudFormation what the values are. For local, you use --env-vars to set the environment variables to a testing value.

If I have a miss understanding of the issue, please create a new issue (closed issues are hard to track).

[whereisaaron]

One item has been resolved 🥳

• Best workaround was a secret/undocumented option (sam deploy --parameter-overrides), this is now documented ✅

The core consistency problem with sam remains:

Right now sam local, sam build, and sam deploy have three, mutually incompatible ways to specify environment variable values.

[ubertao]

Another year passed, sam deploy still can't read env vars in a file.

But here's a workaround. Assuming your project's deployment parameters (e.g. environment variables) are maintained in a .env file like this:

Var1=Value1
VAr2=Value2

You can create an alias like this:

alias samdeploy="sam deploy --parameter-overrides \"`tr -s '\n' ' ' < .env`\""

Not a perfect solution (e.g. easily breaks with an extra space), but better than nothing.

Better yet, if you are working on a nodejs project like me, you can totally use npm to take over development lifecycle.

In package.json:

"scripts": {
   "build": "sam build ...",
   "deploy": "sam deploy --parameter-overrides \"`tr -s '\n' ' ' < .env`\"",
   "invoke": "aws lambda invoke ...",
   "logs": "sam logs ..."
}

[edwardkay]

Building on @ubertao's comment above, I'm also using node / npm. My environment variables are in JSON format in .env/env.json. I have the following in my npm scripts:

  "scripts": {
	"about:start-api": "echo 'Move to the parent directory (where template.yaml lives); remove any compiled code (so we see local changes in real time) and run `sam local start-api`'",
	"start-api": "cd ../ && rm -rf ./.aws-sam && sam local start-api --env-vars .env/env.json",
	"about:deploy": "echo 'Build and deploy. Provide the AWS profile via the command line: npm run deploy --profile=yourProfileName The jq task converts the environment variables in env.json to a set of key=value parameters for the sam deploy command.'",
	"deploy": "cd ../ && rm -rf ./.aws-sam && sam deploy --guided --profile $npm_config_profile --parameter-overrides $(jq -r '.FUNCTION_NAME|to_entries|map(\"ParameterKey=\\(.key),ParameterValue=\\(.value)\") | join(\" \")' .env/env.json)",
	"test": "mocha tests/unit/"
  },

NB: Change FUNCTION_NAME to the name of your Lambda function as defined in your env.json file.

What a faff....