Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Default values for cf references #4940

Closed
walery opened this Issue Apr 27, 2018 · 16 comments

Comments

Projects
None yet
9 participants
@walery
Copy link
Contributor

walery commented Apr 27, 2018

This is a (Bug Report / Feature Proposal)

Using cf references it's not possible to provide default values. Example:

${cf:non-existing-stack.foo, 'bar'}

Description

For bug reports:

  • What went wrong?

Default value was not picked. Instead of this I got error message:

  Serverless Error ---------------------------------------

  Stack with id non-existing-stack does not exist
  • What did you expect should have happened?

Because I've set default value this should be used.

  • What was the config you used?
  • What stacktrace or error message from your provider did you see?

For feature proposals:

  • What is the use case that should be solved. The more detail you describe this in the easier it is to understand for us.
  • If there is additional config how would it look

Similar or dependent issues:

Additional Data

  • Serverless Framework Version you're using: 1.26.1
  • Operating System: darwin
  • Stack Trace:
  • Provider Error messages: Stack with id non-existing-stack does not exist
@kobim

This comment has been minimized.

Copy link
Contributor

kobim commented May 3, 2018

The existing behaviour in case of a non-exported variable from a stack is resulting in error too:

  Serverless Error ---------------------------------------

  Trying to request a non exported variable from CloudFormation. Stack name: "my-prod-stack" Requested variable: "foo".

Is the appropriate fix is to allow default values in both cases, or throw an error?

I like the first approach (not throwing an error), as there's a warning shown if a default parameter isn't specified:

 Serverless Warning --------------------------------------

  A valid CloudFormation stack/variable to satisfy the declaration 'cf:my-prod-stack.foo' could not be found.

What do you think?

@HyperBrain

This comment has been minimized.

Copy link
Member

HyperBrain commented May 3, 2018

From a CI/CD perspective it is in my opinion better to error out than to produce non-working deployments silently. In the worst case a wrongly set variable value might not be noticed immediately in a deployed system and lead to weird behavior that is difficult to debug.
Serverless as a deployment tool cannot know which default value would be appropriate - you do not know what is stored in a CF variable. It could just be anything with any semantical meaning.

@J-Rojas

This comment has been minimized.

Copy link

J-Rojas commented May 21, 2018

I'm having similar trouble when first initializing my deployment. My lambda functions depend on output from a CF stack, but the CF stack is defined in the resources section so the variable is not yet defined on the first deployment. After the first deployment, everything works well.

Is there a best practice for solving this dependency cycle problem?

@kobim

This comment has been minimized.

Copy link
Contributor

kobim commented May 23, 2018

@J-Rojas any reason you refer to Outputs rather than actual Resources of your stack?
Can you share your serverless.yml so I can have a look?

@J-Rojas

This comment has been minimized.

Copy link

J-Rojas commented May 23, 2018

@kobim, I need to reference Outputs generated by AWS CF stack that I don't define directly in my CF template. For example, I need the URL to a RDS instance that is auto-generated after the CF stack is created.

Here's a sample. The environment variables are set to collect the CF stack outputs so they are fed into the Lambda functions. This fails on the first run because the CF stack doesn't exist yet. I found the only way to fix this 'chicken-egg' cycle was to break up the serverless file in two; one part would handle the resources, while the other would handle deploying the lambdas that rely on these resources.

If the CF stack deployment was somehow processed before the lambdas, then my specific problem would not exist.

plugins:
  - serverless-python-requirements
  - serverless-scriptable-plugin
service: database-serverless

custom:
  pythonRequirements:
    dockerizePip: non-linux

  scriptHooks:
    after:aws:deploy:deploy:updateStack: sls invoke -f configure-database

provider:
  name: aws
  runtime: python3.6
  environment:
    DATABASE_URL: ${cf:database-serverless-${opt:stage,'dev'}.DatabaseUrl}
    DATABASE_PORT: ${cf:database-serverless-${opt:stage,'dev'}.DatabasePort}
    DATABASE_USERNAME: dbadmin
    DATABASE_PASSWORD_SSM_KEY: DATABASE_PASSWORD_SSM_KEY
    DATABASE_NAME: db
    DATABASE_NAMES: db_v1
    DATABASE_USERS: db_user
    DATABASE_PASSWORD_KEYS: DB_V1_PASSWORD

  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - "ssm:GetParameter"
      Resource:  arn:aws:ssm:us-west-2:*:parameter/*
    - Effect: "Allow"
      Action:
        - "ssm:PutParameter"
      Resource:  arn:aws:ssm:us-west-2:*:parameter/*

# you can overwrite defaults here
  stage: dev
  region: us-west-2

functions:

  configure-database:
    handler: database.bootstrap_database

resources:
  Resources:

    DBInstance:
      Type: "AWS::RDS::DBInstance"
      Properties:      
        DBInstanceClass: db.t2.small
        Engine: postgres        
        MasterUsername: dbadmin
        MasterUserPassword: "SomePassword"
        DBName: db
        Port: 5432                
        AllocatedStorage: 20
        PreferredBackupWindow: 06:14-06:44
        BackupRetentionPeriod: 6
        DBSecurityGroups: []
        VPCSecurityGroups: [sg-0b345777]
        DBParameterGroupName: encrypted-db          
        AvailabilityZone: us-west-2a
        DBSubnetGroupName: default
        PreferredMaintenanceWindow: sun:13:19-sun:13:49
        MultiAZ: false
        EngineVersion: 9.6.6
        AutoMinorVersionUpgrade: true
        LicenseModel: postgresql-license
        PubliclyAccessible: true
        StorageType: gp2        
        StorageEncrypted: true
        KmsKeyId: arn:aws:kms:us-west-2:*:key/*
        CopyTagsToSnapshot: false
        MonitoringInterval: 0
  Outputs:
    DatabaseUrl:
      Description: The url for the database
      Value:
        "Fn::GetAtt": [ DBInstance, Endpoint.Address ]
    DatabasePort:
      Description: The port for the database
      Value:
        "Fn::GetAtt": [ DBInstance, Endpoint.Port ]
@kobim

This comment has been minimized.

Copy link
Contributor

kobim commented May 23, 2018

You can directly reference the Resource in your environment variables definition:

provider:
  name: aws
  runtime: python3.6
  environment:
    DATABASE_URL:
      "Fn::GetAtt": [ DBInstance, Endpoint.Address ]
    DATABASE_PORT:
      "Fn::GetAtt": [ DBInstance, Endpoint.Port ]

(and even omit the Outputs section)

@J-Rojas

This comment has been minimized.

Copy link

J-Rojas commented May 23, 2018

@kobim that worked! I didn't realize I could use references in the environment section. Thanks for the help.

@ivanbarlog

This comment has been minimized.

Copy link

ivanbarlog commented Jun 14, 2018

@kobim how would you solve the reference for Cognito UserPoolId in ENV variables?

I tried Ref: LogicalNameOfTheUserPool but I get circular reference error. When I am trying the ${cf:myStack-myStage.UserPoolId} where UserPoolId is Output of Ref: LogicalNameOfTheUserPool I get the Stack with id myStackId does not exist.

From the documentation I can read that only Arn can be read via Fn::GetAtt which contains the UserPoolId which I need to reference. Should I just parse it from there or there is some better way to obtain UserPoolId within ENV variables that you can think of?

Thanks for help

@kobim

This comment has been minimized.

Copy link
Contributor

kobim commented Jun 18, 2018

You didn't share your Resources section, but I'll assume it is probably like:

Resources:
  LogicalNameOfTheUserPool:
    Type: 'AWS::Cognito::UserPool'
    Properties:
      UserPoolName: 'My Pool'

hence you can refer to it in the env variables as following:

provider:
  environment:
    USER_POOL:
      Ref: LogicalNameOfTheUserPool

Generally, if you don't use the Outputs of your CloudFormation stack outside the stack, you don't need to define it at all. Just refer to your resources inside serverless.yml where required.

Hope this makes sense.

@ivanbarlog

This comment has been minimized.

Copy link

ivanbarlog commented Jun 18, 2018

Thanks @kobim that was the first thing I have tried but unfortunately that caused me a circular dependency :-(

The CloudFormation template is invalid: Circular dependency between resources: [lots, of, my, resource, logical, names, giving, me, no clue, where, the, circular, dependency, may, be]

@kobim

This comment has been minimized.

Copy link
Contributor

kobim commented Jun 18, 2018

Do you mind sharing the yml file?

@ivanbarlog

This comment has been minimized.

Copy link

ivanbarlog commented Jun 20, 2018

@kobim I would rather not share the config since it is too long and it contains many resources.

I can share simplified version just for you to get the idea:

service:
  name: ${opt:name, 'sample'}
  publish: false # disable auto-publishing to the Serverless platform

plugins:
  - serverless-appsync-plugin
  - serverless-webpack

provider:
  name: aws
  runtime: nodejs8.10
  region: eu-west-1
  stage: ${opt:stage, 'dev'}
  profile: ${opt:aws-profile, 'myProfile'}
  memorySize: 256
  timeout: 15
  versionFunctions: false
  environment:
    # this needs to be commented out before first deployment because CloudFront stack is not known yet
    USER_POOL_ID: ${cf:${self:service}-${self:provider.stage}.UserPoolId}

functions:
  dataResolver:
    handler: handler.dataResolver
    role: AppSyncLambdaRole

  cognito:
    handler: handler.cognito
    role: AppSyncLambdaRole
    events:
      - cognitoUserPool:
          pool: UserPool
          trigger: CustomMessage

      - cognitoUserPool:
          pool: UserPool
          trigger: PostConfirmation

custom:
  accountId:
    Ref: AWS::AccountId
  appSync:
    name: graphql-${self:provider.stage}
    authenticationType: AMAZON_COGNITO_USER_POOLS
    userPoolConfig:
      awsRegion: "${self:provider.region}"
      defaultAction: ALLOW
      userPoolId:
        Ref: CognitoUserPoolUserPool
    mappingTemplatesLocation: mapping-templates
    mappingTemplates:
      # Query
      - dataSource: Lambda_dataResolver
        type: Query
        field: sample
        request: "sample.request"
        response: "default.response"
    schema: schema.graphqls
    serviceRole: "AppSyncServiceRole"
    dataSources:
      - type: AWS_LAMBDA
        name: Lambda_dataResolver
        description: none'
        config:
          lambdaFunctionArn:
            Fn::GetAtt: [DataResolverLambdaFunction, Arn]
          serviceRoleArn:
            Fn::GetAtt: [AppSyncLambdaServiceRole, Arn]

resources:
  # Cognito
  - ${file(resources/cognito-user-pool.yml)}
  - ${file(resources/cognito-identity-pool.yml)}
  # Dynamo
  - ${file(resources/dynamo-table.yml)}
  # Website
  - ${file(resources/s3-website.yml)}
  - ${file(resources/cloudfront-website.yml)}
  # Other
  - ${file(resources/other.yml)}

I think that the circular dependency is caused by serverless-appsync-plugin when it tries to reference to Cognito user pool in custom.appsync.userPoolConfig.userPoolId but I am not sure since I haven't study the code base yet for this.

I think that the best solution for me would be to get rid of that plugin and set up CloudFormation templates manually.

In the meantime, do you have any thoughts about this?

@kobim

This comment has been minimized.

Copy link
Contributor

kobim commented Jun 20, 2018

It sure looks like a serverless-appsync-plugin issue - sid88in/serverless-appsync-plugin#82.

An alternative will be as you said - defining the AppSync resources yourself (that issue has a nice example of that)

@mikecann

This comment has been minimized.

Copy link

mikecann commented Sep 28, 2018

@ivanbarlog @kobim I have exactly the same issue (with userpoolid and circular references) and im not using appsync.

The following produced this error:

  The CloudFormation template is invalid: Circular dependency between resources: [SecretLambdaFunction, HelloLambdaPermissionApiGateway, PreSignUpLambdaFunction, ApiGatewayDeployment1538103844756, SecretLambdaPermissionApiGateway, ApiGatewayMethodHelloGet, MyUsersClient, SecretLambdaVersionOFfqoJDtyCrmcRj0j0QB6PGp0pgv99GQsTs6k1QppI, HelloLambdaFunction, PreSignUpLambdaPermissionCognitoUserPoolMyUserPoolTriggerSourcePreSignUp, PreSignUpLambdaVersion6TuGz1NCl26cdNufjAG9T7N4llBX58Jsjm9miEdsg, HelloLambdaVersionwlFLXuH5zH8AGM9gB3D6nyy4aPgYZAAkkqN5vzKPH0, ApiGatewayMethodSecretPost, CognitoUserPoolMyUserPool]

In serverless.yml:

service:
  name: my-service

plugins:
  - serverless-offline
  - serverless-webpack

custom:
  webpack:
    includeModules: true # enable auto-packing of external modules

provider:
  name: aws
  runtime: nodejs8.10
  stage: ${opt:stage}
  region: ${opt:region, 'ap-southeast-2'}
  memorySize: 256
  environment:
    REGION: ${self:provider.region}
    AWS_USER_POOL_ID:
      Ref: CognitoUserPoolMyUserPool

functions:

  hello:
    handler: src/api/hello/hello.main
    events:
      - http:
          method: get
          path: hello

  secret:
    handler: src/api/secret/secret.main
    events:
      - http:
          method: post
          path: secret
          authorizer: aws_iam

  preSignUp:
    handler: src/triggers/cognito/preSignUp/preSignUp.main
    events:
      - cognitoUserPool:
          pool: MyUserPool
          trigger: PreSignUp

resources:
  - ${file(resources/user-pool.yml)}

in resources/user-pool.yml

Resources:
  # The pool for our users
  CognitoUserPoolMyUserPool:
    Type: AWS::Cognito::UserPool
    Properties:
      UserPoolName: MyUsers_${self:provider.stage}
      EmailVerificationSubject: Your verification code
      SmsAuthenticationMessage: Your authentication code is {####}.
      AutoVerifiedAttributes:
        - phone_number
      LambdaConfig: {}
      AdminCreateUserConfig:
        InviteMessageTemplate:
          EmailMessage: "Your username is {username} and temporary password is
            {####}. "
          EmailSubject: "Your temporary password"
          SMSMessage: "Hi, Your username is {username} and temporary password is
            {####}. "
        UnusedAccountValidityDays: 7
        AllowAdminCreateUserOnly: false
      EmailConfiguration: {}
      AliasAttributes:
        - email
        - phone_number
      Policies:
        PasswordPolicy:
          RequireLowercase: false
          RequireSymbols: false
          RequireNumbers: true
          MinimumLength: 12
          RequireUppercase: false
      UserPoolTags: {}
      SmsVerificationMessage: "Your Account verification code is {####}. "
      EmailVerificationMessage: "Your verification code is {####}. "

  # The client used to connect to the pool
  MyUsersClient:
    Type: AWS::Cognito::UserPoolClient
    Properties:
      ClientName: MyUsersClient_${self:provider.stage}
      UserPoolId:
        Ref: CognitoUserPoolMyUserPool
      ExplicitAuthFlows:
        - ADMIN_NO_SRP_AUTH
      GenerateSecret: false

# Outputs are needed for the "setStage" script to work
Outputs:
  UserPoolId:
    Value:
      Ref: CognitoUserPoolMyUserPool

  UserPoolClientId:
    Value:
      Ref: MyUsersClient

If you dont declare the reference to the userpool in the environment variable then the circular reference issue goes away. The problem is, I need that userpoolId reference in some of my functions..

valscion added a commit to valscion/dissobot that referenced this issue Sep 29, 2018

Document the need to remove deploy hook before first deployment
It's not possible to have CloudFormation stack pointing variables if a
stack has never been created before. Default variables wouldn't work,
either: serverless/serverless#4940
@mikecann

This comment has been minimized.

Copy link

mikecann commented Oct 1, 2018

@kobim I was thinking some more about this over the weekend and I wonder if I am getting circular references because of my use of Cognito UserPool triggers in the function? Perhaps that means you cant use Ref in an environment variable?

@Bob-Thomas

This comment has been minimized.

Copy link

Bob-Thomas commented Oct 9, 2018

@kobim Thanks for the nice trick finally I am not hating that much on cognito in serverless

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.