BREAKING - Separated Packaging and Deployment for CI/CD #3344

Open
wants to merge 1 commit into
from

Projects

None yet

7 participants

@eahefnawy
Member
eahefnawy commented Mar 9, 2017 edited

What did you implement:

Completely separated the packaging and deployment of services/functions into two separate commands/plugins for easier use and control inside CI/CD systems

Closes #2660
Closes #2473

How did you implement it:

Created a new package aws plugin that handles compilation and packaging, and kept the deploy plugin for actually creating/updating and uploading the stack/artifacts. This separation of commands/plugins means that the lifecycle events will break for any plugin author who's hooking into the deploy lifecycle events, because lifecycle events are tied to commands.

How can we verify it:

  • sls deploy => this would package into the default .serverless path and deploy directly afterwards
  • sls package --package <path to package folder> => This would package the service into the provided path
  • sls deploy --package <path to package folder> => This would deploy a package from the provided path

Note A: package path is the path to the folder that contains the zip file along with the core and compiled CF stack templates. So it's not just the path to the artifact zip file because deployment also includes deploying the stack. So the templates are required.

Note B: you can also set the package path in serverless.yml in the package.path property. This property will be used by the package and deploy commands as output/input paths respectively for packaging and deployment. This way you can customize that default .serverless packing folder.

Note C: the --noDeploy option is now dropped since the same functionality is now achieved with the package command.

Todos:

  • Write tests
  • Write documentation
  • Fix linting errors
  • Make sure code coverage hasn't dropped
  • Provide verification config / commands / resources
  • Enable "Allow edits from maintainers" for this PR
  • Update the messages below

Is this ready for review?: YES
Is it a breaking change?: YES

@eahefnawy eahefnawy added this to the 1.10 milestone Mar 9, 2017
@eahefnawy eahefnawy self-assigned this Mar 9, 2017
@eahefnawy eahefnawy requested a review from pmuens Mar 9, 2017
@pmuens
Member
pmuens commented Mar 9, 2017

Nice! Really need to test this thoroughly and look into ways this might affect other provider plugins such as Azure, OpenWhisk or Google...

@HyperBrain
Member

Will the "deploy function" command also use the --package option if set? I do not know if single function deployment is CI/CD relevant at all, so the support there probably can be skipped at all but should be mentioned in the docs.
Any thoughts on that?

@eahefnawy
Member
eahefnawy commented Mar 9, 2017 edited

@HyperBrain excellent question!!! And I ended up with exactly the same conclusion. I think deploy function is meant to quickly update function code and not in CI/CD, so right now deploy function handles packaging the function as well.

@johncmckim
Member

@eahefnawy if you'd like some help testing I would be very happy to try out this PR. If so, let me know when it's ready to test.

'deploy:deploy': () => BbPromise.bind(this)
- .then(this.mergeCustomProviderResources)
+ .then(this.createStack)
@HyperBrain
HyperBrain Mar 12, 2017 Member

This could be combined with the entry point PR to have the deploy split up in more fine-grained hookable lifecycle events.

module.exports = {
uploadCloudFormationFile() {
this.serverless.cli.log('Uploading CloudFormation file to S3...');
- const body = JSON.stringify(this.serverless.service.provider.compiledCloudFormationTemplate);
@HyperBrain
HyperBrain Mar 12, 2017 edited Member

It would be important to keep the compiled template under the provider object. E.g. the alias plugin needs that information to create the alias stack information and modifies the compiled template. So it accesses this.serverless.service.provider.compiledCloudFormationTemplate to extract its information and modify the template in-memory. If the template is made local to a component, it will break all plugins that modify the template before it gets further processed by the AWSDeploy plugin.

The package command (including awsPackage) should spawn some inner lifecycles:

-> package
  ->package:prepare
    -> awsPackage:prepare
  -> package:compile
    -> package:compileFunctions
    -> package:compileTemplate
      -> awsPackage:compileTemplate
      -> awsPackage:mergeIamTemplates
      -> awsPackage:mergeUserResources
  -> package:persistArtifacts

And keep the compiled template under provider until the package:persistArtifacts. Then plugins can hook after or before package:compileTemplate. The awsPackage entrypoint would hook the package lifecycle events.

BTW: The lifecycle list above is just a quickly written down example - just to get an idea what I mean.

I could create a further PR based on your PR/branch that would extend and arrange the lifecycles in a usable way utilizing the entrypoint PR's spawn function. Just let me know.

@laardee laardee referenced this pull request in SC5/serverless-deployment-codepipeline Mar 12, 2017
Open

Serverless packaging for CI/CD #4

@pmuens

Just gave it a first spin.

Really like the separation of package and deploy and the way it works together when I only run serverless deploy 👍

Super convenient!

It seems like the .serverless dir is removed when I run serverless package --package foo which surprised me. I as a user would assume that the package command is nont destructive.

Furthermore I did a global search for options.noDeploy and noDeploy. Some old references are still around and should be removed before merging.

I'll review / test it more in depth this week!

@HyperBrain
Member
HyperBrain commented Mar 14, 2017 edited

@pmuens I will add a PR to the package-plugin branch as soon as the ep PR has been merged to change/set the lifecycle model in a convienient way. Additionally we would change it a bit to maximize code and lifecycle reuse. Had some discussions already with @eahefnawy on Slack about that.

@stevecaldwell77
Contributor

lifecycle events will break for any plugin author who's hooking into the deploy lifecycle events, because lifecycle events are tied to commands.

Can this be explained with a little more detail? For instance, consider the following plugin:

  • hooks into 'before:deploy:deploy'
  • applies some transformations to the resources section

Will that still run correctly during sls deploy?

Seeing as how such a plugin is really concerned with packaging, should it be refactored to hook into a packaging event?

If it is hooked to a packaging event, will it still run during sls deploy? And not run during sls deploy --package foo?

@HyperBrain
Member
HyperBrain commented Mar 15, 2017 edited

@stevecaldwell77 The goal is to have the old events still available, so nothing would break for plugin authors. Instead it would be so, that new sub-lifecycles would be spawned that allow even more fine-grained control for plugin writers for future plugin versions, as the current ones are really coarse.

You can have a look at my recent blog post, on how to optimize and expose lifecycles in a plugin to get a picture how this also works for plugins (https://serverless.com/blog/advanced-plugin-development-extending-the-core-lifecycle/).

I am pretty sure that the solution here will be non breaking for existing plugins but will improve further versions a lot.

BTW: I think the "lifecycle events will break for any plugin author who's hooking into the deploy lifecycle events, because lifecycle events are tied to commands." is outdated information already.

@stevecaldwell77
Contributor

@HyperBrain - great news, thanks for the detailed response. Nice blog post too.

@eahefnawy eahefnawy seperated packaging and deployment
ac9ee65
@ryan-dyer-sp

Any possibility of the file path supporting an s3 path?

@horike37
Member

@eahefnawy @pmuens
I'm implementing the cloudwatch logs event feature.
Is there anything I need to be careful of in the relation to this issue?

@pmuens
Member
pmuens commented Mar 23, 2017

@horike37 nice! Thanks for jumping on this one!

Is there anything I need to be careful of in the relation to this issue?

Not quite sure how this might impact this PR. Any thoughts @eahefnawy ?

@pmuens pmuens referenced this pull request Mar 23, 2017
Closed

Allow the package artifact to contain wildcards. #3298

7 of 7 tasks complete
@eahefnawy
Member

@horike37 not sure about the scope of your implementation, but this PR only changes packing and deployment and their relevant lifecycle events. Anything that happens after the functions are deployed (invoking/logging...etc) should stay the same way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment