Add full custom resource merging support

[pmuens]

This PR implements the full support for custom resource merging as discussed in #1833

/cc @flomotlik @eahefnawy

[flomotlik]

Would add a Description here as well. Its optional, but if people see it in the docs they know its available

[flomotlik]

@pmuens could you also add an example here as a comment with a service that overwrites a resource that we create to validate this. So not just overwriting like with an output or Description, but merging in additional config items

[pmuens]

@flomotlik here's an example which merges in a new description for the ServerlessDeploymentBucketName output variable:

service: my-service # NOTE: update this with your service name

provider:
  name: aws
  runtime: nodejs4.3

functions:
  hello:
    handler: handler.hello
    events:
      - http:
          path: hello
          method: get
      - s3: my-new-shiny-serverless-bucket
  world:
    handler: handler.world

# you can add CloudFormation resource templates here
resources:
  Outputs:
    ServerlessDeploymentBucketName:
      Description: 'Description for Serverless deployment bucket'

We're not able to overwrite CloudFormation resources which are created with the events syntax as those definitions are applied in the deploy:deploy lifecycle which appears after deploy:setupProviderConfiguration which we use to merge in custom provider resources.

I've also added a test which overwrites the ServerlessDeploymentBucket resources (and merges in additional stuff) which should cover your request above:

serverless/lib/plugins/aws/deploy/tests/createStack.js

Lines 186 to 205 in 121df4d

	it('should be able to overwrite existing object properties', () => {
	const customResourcesMock = {
	Resources: {
	ServerlessDeploymentBucket: {
	Type: 'Some::New::Type',
	FakeResource1: 'FakePropValue',
	FakeResource2: {
	FakePropKey: 'FakePropValue',
	},
	},
	},
	};
	awsDeploy.serverless.service.resources = customResourcesMock;
	return awsDeploy.createStack().then(() => {
	expect(awsDeploy.serverless.service.resources.Resources.ServerlessDeploymentBucket)
	.to.deep.equal(customResourcesMock.Resources.ServerlessDeploymentBucket);
	});
	});

[flomotlik]

We're not able to overwrite CloudFormation resources which are created with the events syntax as those definitions are applied in the deploy:deploy lifecycle which appears after deploy:setupProviderConfiguration which we use to merge in custom provider resources

Then we should change that. We definitely have to support overwriting anything we create through custom resources. Otherwise people simply won't be able to do it at all. The only real solution I see is merging it back in during before:deploy:deploy.

[pmuens]

@flomotlik just pushed the refactoring. Now all the stuff is merged into the serverless.service.provider.compiledCloudFormationTemplate object.

Overwriting the whole template is as easy as using simply the resources section in the serverless.yml file.

Here's an example service file which shows how this can be done. A DisplayName property for the SNS event is added with the help of the resources section. Furthermore the Description of the ServerlessDeplymentBucketName output variable is set.

I've added all available events so that we validate that everything works (which does on my machine and the integration tests).

# Welcome to Serverless!
#
# This file is the main config file for your service.
# It's very minimal at this point and uses default values.
# You can always add more config options for more control.
# We've included some commented out config examples here.
# Just uncomment any of them to get that config option.
#
# For full config options, check the docs:
#    v1.docs.serverless.com
#
# Happy Coding!

service: my-service # NOTE: update this with your service name

provider:
  name: aws
  runtime: nodejs4.3

# you can overwrite defaults here
#defaults:
#  stage: dev
#  region: us-east-1

# you can add packaging information here
#package:
#  include:
#    - include-me.js
#  exclude:
#    - exclude-me.js
#  artifact: my-service-code.zip

functions:
  hello:
    handler: handler.hello
    events:
      - http:
          path: hello
          method: get
      - s3: my-new-shiny-serverless-bucket
  world:
    handler: handler.world
    events:
      - sns: some-random-topic-name
      - schedule: rate(1 minute)


#   you can add any of the following events
#   events:
#     - http:
#         path: users/create
#         method: get
#     - s3: ${bucket}
#     - schedule: rate(10 minutes)
#     - sns: greeter-topic

# you can add CloudFormation resource templates here
resources:
  Resources:
    worldSNSEvent0:
      Properties:
        DisplayName: 'Description for SNS topic'
  Outputs:
    ServerlessDeploymentBucketName:
      Description: 'Description for Serverless deployment bucket'

[flomotlik]

Don't think this line is necessary any more as the core template will create Outputs already

[flomotlik]

I tested it with all the above mentioned Events (http, S3, SNS, Schedule) and it worked with all of them.

One thing that does fail though if the resources.Resources section is empty:

resources:
  Resources:
  Outputs:
    ServerlessDeploymentBucketName:
      Description: 'Description for Serverless deployment bucket'

Error Message:

Serverless Error ---------------------------------------

     'null' values are not allowed in templates

  Get Support --------------------------------------------
     Docs:          v1.docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues

Looks like its simply replacing the whole Resources section with null in that case. Probably best to set Resources and outputs to empty hashes in case they are null. Or we simply properly document it but this could imho be a bad experience.

[pmuens]

Yep sounds good (I would do both, setting it to empty hashes and document it).

[nicka]

Looks nice guys!!! Not completely sure about the world -> worldSNSEvent0 conversion. In some cases we have like 20 similar events for S3 with different suffix/prefix. This would require a lot of "thinking" to get it to map correctly with your defined CloudFormation resources(looking up 0..20 and debugging events).

Some food for thought:

functions:
  hello:
    handler: handler.hello
    events:
      - hello-get:
          type: http
          config:
            path: hello
            method: get
      - hello-head:
          type: http
          config:
            path: hello
            method: head
      - hello-s3-jpg
          type: s3
          config:
            bucket: HelloS3Bucket,
            bucketEvents: 
              - s3:ObjectCreated
            filterRules:
              - name: suffix
                value: jpg
              - name: prefix
                value: foo/
      - hello-s3-bar
          type: s3
          config:
            bucket: HelloS3Bucket,
            bucketEvents: 
              - s3:ObjectCreated
            filterRules:
              - name: suffix
                value: png
              - name: prefix
                value: bar/
      - hello-sns
          type: sns
      - hello-schedule
          type: schedule
          config:
            schedule: rate(1 minute)

### If you want to overwrite/add extra information
resources:
  Resources:
    HelloS3Bucket:
      Type: AWS::S3::Bucket
    HelloS3Jpg:
      Type: AWS::S3::Bucket
      Properties:
        Description: 'Description for S3 event'
    HelloSns:
      Properties:
        DisplayName: 'Description for SNS topic'
    # Etc...

Or short-hand with logical-id but would require more support per service and key/value mapping:

functions:
  hello:
    handler: handler.hello
    events:
      - http:
          path: hello
          method: get
      - http:
          path: hello
          method: head
      - s3:
          logical-id: hello-s3-jpg
          bucket: HelloS3Bucket,
          bucketEvents: 
            - s3:ObjectCreated
          filterRules:
            - name: suffix
              value: jpg
            - name: prefix
              value: foo/
      - s3:
          logical-id: hello-s3-png
          bucket: HelloS3Bucket,
          bucketEvents: 
            - s3:ObjectCreated
          filterRules:
            - name: suffix
              value: png
            - name: prefix
              value: bar/
      - sns:
          logical-id: hello-sns
      - schedule:
          logical-id: hello-schedule
          schedule: rate(1 minute)

### If you want to overwrite/add extra information
resources:
  Resources:
    HelloS3Bucket:
      Type: AWS::S3::Bucket
    HelloS3Jpg:
      Type: AWS::S3::Bucket
      Properties:
        Description: 'Description for S3 event'
    HelloSns:
      Properties:
        DisplayName: 'Description for SNS topic'
    # Etc...

[flomotlik]

@nicka we have to change the naming of resources. At the moment we have some inconsistencies there, e.g. the SNS topic doesn't have the SNS topic name in the logical name for example. Other parts (like S3) get a number added at the end, ....

This needs to be cleaned up before a next release so its consistent and documented. I like the idea of adding logical-id as a potential option to events so it can be set directly (although sometimes one event could mean multiple CF resources, so one name might not be enough)

[flomotlik]

@pmuens could you put the fix for the resources in place, then I think this is ready to be merged

[pmuens]

@flomotlik done

[pmuens]

Updated the last commit with a quick fix should be ready now.

[nicka]

🚀