New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include Multiple Resource Files #2828

Closed
kurtmaile opened this Issue Nov 30, 2016 · 13 comments

Comments

Projects
None yet
8 participants
@kurtmaile

kurtmaile commented Nov 30, 2016

This is a Feature Proposal

Description

This feature enables multiple resource files to be included in the 'Resources' section. This enables larger services to include sub resource files independantly. One resource file can be broken up a number to enable better independant managability / change control as these files can get large and hard to manage in larger implementations.

e.g

resources:
  Resources:
    - ${file(resources/first-cf-resources.yml)}
    - ${file(resources/second-cf-resources.yml)}

@pmuens pmuens added the kind/feature label Nov 30, 2016

@dougmoscrop dougmoscrop referenced this issue Dec 2, 2016

Merged

add Support for resources as an array #2842

7 of 7 tasks complete

@pmuens pmuens added this to the 1.5 milestone Dec 7, 2016

@pmuens

This comment has been minimized.

Show comment
Hide comment
@pmuens

pmuens Dec 7, 2016

Member

I ❤️ this one! Thanks for bringing this up!

Member

pmuens commented Dec 7, 2016

I ❤️ this one! Thanks for bringing this up!

@pmuens pmuens self-assigned this Dec 7, 2016

@pmuens pmuens modified the milestones: 1.5, 1.4 Dec 7, 2016

@nikgraf

This comment has been minimized.

Show comment
Hide comment
@nikgraf

nikgraf Dec 7, 2016

Contributor

Not sure about this one. What happens in case there are conflicts since cloud formation is key based? Do we validate them or just let it fail at a deploy.

In addition this moves us away from the default CloudFormation standard.

@eahefnawy @mthenw thoughts?

Contributor

nikgraf commented Dec 7, 2016

Not sure about this one. What happens in case there are conflicts since cloud formation is key based? Do we validate them or just let it fail at a deploy.

In addition this moves us away from the default CloudFormation standard.

@eahefnawy @mthenw thoughts?

@pmuens

This comment has been minimized.

Show comment
Hide comment
@pmuens

pmuens Dec 7, 2016

Member

AFAIK @dougmoscrop implementation considers everything in resources and not just resources.Resources. So it's way more flexible.

The use case @kurtmaile showed here is just one out of many.

I think of this in a more multi provider way. Right now we don't support arrays on the resources level. This might be a good addition for others providers who want to integrate (as they might want to store smth. in resources in a non-object way). Furthermore it's just an addition / a tiny feature which makes everything more flexible w/o breaking existing stuff.

Member

pmuens commented Dec 7, 2016

AFAIK @dougmoscrop implementation considers everything in resources and not just resources.Resources. So it's way more flexible.

The use case @kurtmaile showed here is just one out of many.

I think of this in a more multi provider way. Right now we don't support arrays on the resources level. This might be a good addition for others providers who want to integrate (as they might want to store smth. in resources in a non-object way). Furthermore it's just an addition / a tiny feature which makes everything more flexible w/o breaking existing stuff.

@pmuens pmuens closed this in #2842 Dec 8, 2016

@kevindiamond

This comment has been minimized.

Show comment
Hide comment
@kevindiamond

kevindiamond Mar 16, 2017

Is there an example of how to use this properly as I tried using the proposal at the top and it fails to render correctly.

kevindiamond commented Mar 16, 2017

Is there an example of how to use this properly as I tried using the proposal at the top and it fails to render correctly.

@pmuens

This comment has been minimized.

Show comment
Hide comment
@pmuens

pmuens Mar 17, 2017

Member

Hey @kevindiamond
how does your serverless.yml and the corresponding .yml files you want to include look like?

What error message do you get?

Member

pmuens commented Mar 17, 2017

Hey @kevindiamond
how does your serverless.yml and the corresponding .yml files you want to include look like?

What error message do you get?

@kevindiamond

This comment has been minimized.

Show comment
Hide comment
@kevindiamond

kevindiamond Mar 17, 2017

So I was trying to do this in the resources section and attempted a handful of different formats including the example at the top of this.

resources:
  Resources:
    - ${file(./pipeline.yml)}
    - DynamoTable:

Was my first attempt to do both a file and then some inline resources. I tried moving "Resources" into each section, that didn't work either.

resources:
  - ${file(./pipeline.yml)}
  - Resources:
      DynamoTable:

I would either get it trying to put an array object into Cloudformation or splitting my ${file(./pipeline.yml)} into 22 objects each with one character in it.

So just looking for any examples on how this works correctly.

Thanks!

kevindiamond commented Mar 17, 2017

So I was trying to do this in the resources section and attempted a handful of different formats including the example at the top of this.

resources:
  Resources:
    - ${file(./pipeline.yml)}
    - DynamoTable:

Was my first attempt to do both a file and then some inline resources. I tried moving "Resources" into each section, that didn't work either.

resources:
  - ${file(./pipeline.yml)}
  - Resources:
      DynamoTable:

I would either get it trying to put an array object into Cloudformation or splitting my ${file(./pipeline.yml)} into 22 objects each with one character in it.

So just looking for any examples on how this works correctly.

Thanks!

@pmuens

This comment has been minimized.

Show comment
Hide comment
@pmuens

pmuens Mar 20, 2017

Member

Thanks for the response @kevindiamond

Importing .json files as a whole object should be possible: https://serverless.com/framework/docs/providers/aws/guide/variables#multiple-configuration-files

There was a PR the other day which extended the import feature: #3378

However it was closed to reduce technical debt.

Member

pmuens commented Mar 20, 2017

Thanks for the response @kevindiamond

Importing .json files as a whole object should be possible: https://serverless.com/framework/docs/providers/aws/guide/variables#multiple-configuration-files

There was a PR the other day which extended the import feature: #3378

However it was closed to reduce technical debt.

@ubaniabalogun

This comment has been minimized.

Show comment
Hide comment
@ubaniabalogun

ubaniabalogun Apr 21, 2017

Hey, @pmuens.
Thanks for your attention on this issue.

I'm running into the same problem @kevindiamond documented in his last comment (see here).

Are we sure there isn't a bug in here?

Here's what my application structure looks like

.
├── resource_configs
│   └── dynamodb.yml
└── serverless.yml

And my serverless.yml

service: service-name
provider:
  name: aws
  runtime: python2.7
resources:
  - ${file(../resource_configs/dynamodb.yml):Resources}

Note that I've also tried ${file(./resource_configs/dynamodb.yml:Resources)} and ${file(resource_configs/dynamodb.yml:Resources)}

Instead of loading my config file, however, the framework is splitting up the string ${file(./resource_configs/dynamodb.yml:Resources)} into individual characters and adding it to my Cloudformation template as such

{
  "0": "$",
  "1": "{",
  "2": "f",
  "3": "i",
  "4": "l",
  "5": "e",
  "6": "(",
  "7": "r",
  "8": "e",
  "9": "s",
  "10": "o",
  "11": "u",
  "12": "r",
  "13": "c",
  "14": "e",
  "15": "_",
  "16": "c",
  "17": "o",
  "18": "n",
  "19": "f",
  "20": "i",
  "21": "g",
  "22": "s",
  "23": "/",
  "24": "d",
  "25": "y",
  "26": "n",
  "27": "a",
  "28": "m",
  "29": "o",
  "30": "d",
  "31": "b",
  "32": ".",
  "33": "y",
  "34": "m",
  "35": "l",
  "36": ")",
  "37": ":",
  "38": "R",
  "39": "e",
  "40": "s",
  "41": "o",
  "42": "u",
  "43": "r",
  "44": "c",
  "45": "e",
  "46": "s",
  "47": "}",
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "The AWS CloudFormation template for this Serverless application",
  "Resources": {
    "ServerlessDeploymentBucket": {
      "Type": "AWS::S3::Bucket"
    }
  },
  "Outputs": {
    "ServerlessDeploymentBucketName": {
      "Value": {
        "Ref": "ServerlessDeploymentBucket"
      }
    }
  }
}

Can you take a second look at this to confirm the feature works as expected?
If it does, can you please provide a definitive example on how to use the feature?
If not, please let me know and I'll document my findings in a bug ticket.

Thanks,

ubaniabalogun commented Apr 21, 2017

Hey, @pmuens.
Thanks for your attention on this issue.

I'm running into the same problem @kevindiamond documented in his last comment (see here).

Are we sure there isn't a bug in here?

Here's what my application structure looks like

.
├── resource_configs
│   └── dynamodb.yml
└── serverless.yml

And my serverless.yml

service: service-name
provider:
  name: aws
  runtime: python2.7
resources:
  - ${file(../resource_configs/dynamodb.yml):Resources}

Note that I've also tried ${file(./resource_configs/dynamodb.yml:Resources)} and ${file(resource_configs/dynamodb.yml:Resources)}

Instead of loading my config file, however, the framework is splitting up the string ${file(./resource_configs/dynamodb.yml:Resources)} into individual characters and adding it to my Cloudformation template as such

{
  "0": "$",
  "1": "{",
  "2": "f",
  "3": "i",
  "4": "l",
  "5": "e",
  "6": "(",
  "7": "r",
  "8": "e",
  "9": "s",
  "10": "o",
  "11": "u",
  "12": "r",
  "13": "c",
  "14": "e",
  "15": "_",
  "16": "c",
  "17": "o",
  "18": "n",
  "19": "f",
  "20": "i",
  "21": "g",
  "22": "s",
  "23": "/",
  "24": "d",
  "25": "y",
  "26": "n",
  "27": "a",
  "28": "m",
  "29": "o",
  "30": "d",
  "31": "b",
  "32": ".",
  "33": "y",
  "34": "m",
  "35": "l",
  "36": ")",
  "37": ":",
  "38": "R",
  "39": "e",
  "40": "s",
  "41": "o",
  "42": "u",
  "43": "r",
  "44": "c",
  "45": "e",
  "46": "s",
  "47": "}",
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "The AWS CloudFormation template for this Serverless application",
  "Resources": {
    "ServerlessDeploymentBucket": {
      "Type": "AWS::S3::Bucket"
    }
  },
  "Outputs": {
    "ServerlessDeploymentBucketName": {
      "Value": {
        "Ref": "ServerlessDeploymentBucket"
      }
    }
  }
}

Can you take a second look at this to confirm the feature works as expected?
If it does, can you please provide a definitive example on how to use the feature?
If not, please let me know and I'll document my findings in a bug ticket.

Thanks,

@pmuens

This comment has been minimized.

Show comment
Hide comment
@pmuens

pmuens Apr 24, 2017

Member

@ubaniabalogun thanks for getting back on this issue. 👍

I tried it locally and was able to reproduce this. Seems like it's a bug. Would be nice if you could file a bug issue. Thanks in advance!

Member

pmuens commented Apr 24, 2017

@ubaniabalogun thanks for getting back on this issue. 👍

I tried it locally and was able to reproduce this. Seems like it's a bug. Would be nice if you could file a bug issue. Thanks in advance!

@jlamandecap

This comment has been minimized.

Show comment
Hide comment
@jlamandecap

jlamandecap Sep 30, 2017

Hi,
Using serverless version 1.23.0

Sorry but I can't get it working with this syntax :

resources:
  Resources:
    - ${file(resources/first-cf-resources.yml)}
    - ${file(resources/second-cf-resources.yml)}

The CF JSON template generated includes contents of my external files but the "Resources" object is an array :

"Resources": [
    {
         "Resource1":...
    },
    ...
]

So CloudFormation complains about this structure :

The CloudFormation template is invalid: Template format error: Any Resources member must be an object.

Is this a regression ?

Thanks

jlamandecap commented Sep 30, 2017

Hi,
Using serverless version 1.23.0

Sorry but I can't get it working with this syntax :

resources:
  Resources:
    - ${file(resources/first-cf-resources.yml)}
    - ${file(resources/second-cf-resources.yml)}

The CF JSON template generated includes contents of my external files but the "Resources" object is an array :

"Resources": [
    {
         "Resource1":...
    },
    ...
]

So CloudFormation complains about this structure :

The CloudFormation template is invalid: Template format error: Any Resources member must be an object.

Is this a regression ?

Thanks

@HyperBrain

This comment has been minimized.

Show comment
Hide comment
@HyperBrain

HyperBrain Sep 30, 2017

Member

Hi @jlamandecap , thanks for asking 👍 .

According to your definition:

resources:
  Resources:
    - ${file(resources/first-cf-resources.yml)}
    - ${file(resources/second-cf-resources.yml)}

You declare Resources as array with the contents of the 2 files as members. So what you see in the template exactly resembles the definition in your yaml file.

@pmuens Is there currently a way to specify multiple file imports without using an array (which leads to the behavior mentioned above)?

Member

HyperBrain commented Sep 30, 2017

Hi @jlamandecap , thanks for asking 👍 .

According to your definition:

resources:
  Resources:
    - ${file(resources/first-cf-resources.yml)}
    - ${file(resources/second-cf-resources.yml)}

You declare Resources as array with the contents of the 2 files as members. So what you see in the template exactly resembles the definition in your yaml file.

@pmuens Is there currently a way to specify multiple file imports without using an array (which leads to the behavior mentioned above)?

@jlamandecap

This comment has been minimized.

Show comment
Hide comment
@jlamandecap

jlamandecap Sep 30, 2017

Hi
thanks @HyperBrain for looking at my problem.

I found a way to use different Resources files :

resources:
   - ${file(resources/first-cf-resources.yml)}
   - ${file(resources/first-cf-resources.yml)}

Each of the yml "included" must start with a "Resources" Object :

---
Resources:
  MyCFResource:
    Type:.....

that's a bit different from this proposal but it works.

jlamandecap commented Sep 30, 2017

Hi
thanks @HyperBrain for looking at my problem.

I found a way to use different Resources files :

resources:
   - ${file(resources/first-cf-resources.yml)}
   - ${file(resources/first-cf-resources.yml)}

Each of the yml "included" must start with a "Resources" Object :

---
Resources:
  MyCFResource:
    Type:.....

that's a bit different from this proposal but it works.

@philiiiiiipp

This comment has been minimized.

Show comment
Hide comment
@philiiiiiipp

philiiiiiipp Aug 24, 2018

I want to add that there is the possibility to have "in file" and external resources / outputs mixed, like so:

resources:
  - Resource:
      ApiGatewayRestApi:
        Type: AWS::ApiGateway::RestApi

  - ${file(resources/first-cf-resources.yml)}
  - ${file(resources/second-cf-resources.yml)}

  - Outputs:
      CognitoUserPoolId:
      Value:
        Ref: CognitoUserPool

philiiiiiipp commented Aug 24, 2018

I want to add that there is the possibility to have "in file" and external resources / outputs mixed, like so:

resources:
  - Resource:
      ApiGatewayRestApi:
        Type: AWS::ApiGateway::RestApi

  - ${file(resources/first-cf-resources.yml)}
  - ${file(resources/second-cf-resources.yml)}

  - Outputs:
      CognitoUserPoolId:
      Value:
        Ref: CognitoUserPool
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment