Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include Multiple Resource Files #2828

Closed
kurtmaile opened this issue Nov 30, 2016 · 13 comments
Closed

Include Multiple Resource Files #2828

kurtmaile opened this issue Nov 30, 2016 · 13 comments
Assignees
Labels
Milestone

Comments

@kurtmaile
Copy link

@kurtmaile kurtmaile commented Nov 30, 2016

This is a Feature Proposal

Description

This feature enables multiple resource files to be included in the 'Resources' section. This enables larger services to include sub resource files independantly. One resource file can be broken up a number to enable better independant managability / change control as these files can get large and hard to manage in larger implementations.

e.g

resources:
  Resources:
    - ${file(resources/first-cf-resources.yml)}
    - ${file(resources/second-cf-resources.yml)}
@pmuens pmuens added the feature label Nov 30, 2016
@dougmoscrop dougmoscrop mentioned this issue Dec 2, 2016
7 of 7 tasks complete
@pmuens pmuens added this to the 1.5 milestone Dec 7, 2016
@pmuens
Copy link
Contributor

@pmuens pmuens commented Dec 7, 2016

I ❤️ this one! Thanks for bringing this up!

@pmuens pmuens self-assigned this Dec 7, 2016
@pmuens pmuens modified the milestones: 1.5, 1.4 Dec 7, 2016
@nikgraf
Copy link
Contributor

@nikgraf nikgraf commented Dec 7, 2016

Not sure about this one. What happens in case there are conflicts since cloud formation is key based? Do we validate them or just let it fail at a deploy.

In addition this moves us away from the default CloudFormation standard.

@eahefnawy @mthenw thoughts?

@pmuens
Copy link
Contributor

@pmuens pmuens commented Dec 7, 2016

AFAIK @dougmoscrop implementation considers everything in resources and not just resources.Resources. So it's way more flexible.

The use case @kurtmaile showed here is just one out of many.

I think of this in a more multi provider way. Right now we don't support arrays on the resources level. This might be a good addition for others providers who want to integrate (as they might want to store smth. in resources in a non-object way). Furthermore it's just an addition / a tiny feature which makes everything more flexible w/o breaking existing stuff.

@pmuens pmuens closed this in #2842 Dec 8, 2016
@kevindiamond
Copy link

@kevindiamond kevindiamond commented Mar 16, 2017

Is there an example of how to use this properly as I tried using the proposal at the top and it fails to render correctly.

@pmuens
Copy link
Contributor

@pmuens pmuens commented Mar 17, 2017

Hey @kevindiamond
how does your serverless.yml and the corresponding .yml files you want to include look like?

What error message do you get?

@kevindiamond
Copy link

@kevindiamond kevindiamond commented Mar 17, 2017

So I was trying to do this in the resources section and attempted a handful of different formats including the example at the top of this.

resources:
  Resources:
    - ${file(./pipeline.yml)}
    - DynamoTable:

Was my first attempt to do both a file and then some inline resources. I tried moving "Resources" into each section, that didn't work either.

resources:
  - ${file(./pipeline.yml)}
  - Resources:
      DynamoTable:

I would either get it trying to put an array object into Cloudformation or splitting my ${file(./pipeline.yml)} into 22 objects each with one character in it.

So just looking for any examples on how this works correctly.

Thanks!

@pmuens
Copy link
Contributor

@pmuens pmuens commented Mar 20, 2017

Thanks for the response @kevindiamond

Importing .json files as a whole object should be possible: https://serverless.com/framework/docs/providers/aws/guide/variables#multiple-configuration-files

There was a PR the other day which extended the import feature: #3378

However it was closed to reduce technical debt.

@ubaniabalogun
Copy link

@ubaniabalogun ubaniabalogun commented Apr 21, 2017

Hey, @pmuens.
Thanks for your attention on this issue.

I'm running into the same problem @kevindiamond documented in his last comment (see here).

Are we sure there isn't a bug in here?

Here's what my application structure looks like

.
├── resource_configs
│   └── dynamodb.yml
└── serverless.yml

And my serverless.yml

service: service-name
provider:
  name: aws
  runtime: python2.7
resources:
  - ${file(../resource_configs/dynamodb.yml):Resources}

Note that I've also tried ${file(./resource_configs/dynamodb.yml:Resources)} and ${file(resource_configs/dynamodb.yml:Resources)}

Instead of loading my config file, however, the framework is splitting up the string ${file(./resource_configs/dynamodb.yml:Resources)} into individual characters and adding it to my Cloudformation template as such

{
  "0": "$",
  "1": "{",
  "2": "f",
  "3": "i",
  "4": "l",
  "5": "e",
  "6": "(",
  "7": "r",
  "8": "e",
  "9": "s",
  "10": "o",
  "11": "u",
  "12": "r",
  "13": "c",
  "14": "e",
  "15": "_",
  "16": "c",
  "17": "o",
  "18": "n",
  "19": "f",
  "20": "i",
  "21": "g",
  "22": "s",
  "23": "/",
  "24": "d",
  "25": "y",
  "26": "n",
  "27": "a",
  "28": "m",
  "29": "o",
  "30": "d",
  "31": "b",
  "32": ".",
  "33": "y",
  "34": "m",
  "35": "l",
  "36": ")",
  "37": ":",
  "38": "R",
  "39": "e",
  "40": "s",
  "41": "o",
  "42": "u",
  "43": "r",
  "44": "c",
  "45": "e",
  "46": "s",
  "47": "}",
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "The AWS CloudFormation template for this Serverless application",
  "Resources": {
    "ServerlessDeploymentBucket": {
      "Type": "AWS::S3::Bucket"
    }
  },
  "Outputs": {
    "ServerlessDeploymentBucketName": {
      "Value": {
        "Ref": "ServerlessDeploymentBucket"
      }
    }
  }
}

Can you take a second look at this to confirm the feature works as expected?
If it does, can you please provide a definitive example on how to use the feature?
If not, please let me know and I'll document my findings in a bug ticket.

Thanks,

@pmuens
Copy link
Contributor

@pmuens pmuens commented Apr 24, 2017

@ubaniabalogun thanks for getting back on this issue. 👍

I tried it locally and was able to reproduce this. Seems like it's a bug. Would be nice if you could file a bug issue. Thanks in advance!

@jlamandecap
Copy link

@jlamandecap jlamandecap commented Sep 30, 2017

Hi,
Using serverless version 1.23.0

Sorry but I can't get it working with this syntax :

resources:
  Resources:
    - ${file(resources/first-cf-resources.yml)}
    - ${file(resources/second-cf-resources.yml)}

The CF JSON template generated includes contents of my external files but the "Resources" object is an array :

"Resources": [
    {
         "Resource1":...
    },
    ...
]

So CloudFormation complains about this structure :

The CloudFormation template is invalid: Template format error: Any Resources member must be an object.

Is this a regression ?

Thanks

@HyperBrain
Copy link
Member

@HyperBrain HyperBrain commented Sep 30, 2017

Hi @jlamandecap , thanks for asking 👍 .

According to your definition:

resources:
  Resources:
    - ${file(resources/first-cf-resources.yml)}
    - ${file(resources/second-cf-resources.yml)}

You declare Resources as array with the contents of the 2 files as members. So what you see in the template exactly resembles the definition in your yaml file.

@pmuens Is there currently a way to specify multiple file imports without using an array (which leads to the behavior mentioned above)?

@jlamandecap
Copy link

@jlamandecap jlamandecap commented Sep 30, 2017

Hi
thanks @HyperBrain for looking at my problem.

I found a way to use different Resources files :

resources:
   - ${file(resources/first-cf-resources.yml)}
   - ${file(resources/first-cf-resources.yml)}

Each of the yml "included" must start with a "Resources" Object :

---
Resources:
  MyCFResource:
    Type:.....

that's a bit different from this proposal but it works.

@philiiiiiipp
Copy link
Contributor

@philiiiiiipp philiiiiiipp commented Aug 24, 2018

I want to add that there is the possibility to have "in file" and external resources / outputs mixed, like so:

resources:
  - Resource:
      ApiGatewayRestApi:
        Type: AWS::ApiGateway::RestApi

  - ${file(resources/first-cf-resources.yml)}
  - ${file(resources/second-cf-resources.yml)}

  - Outputs:
      CognitoUserPoolId:
      Value:
        Ref: CognitoUserPool
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
You can’t perform that action at this time.