feat: decouple "synth" and "deploy" through cloud assemblies #2636

eladb · 2019-05-24T21:04:40Z

Formalize the concept of a cloud assembly to allow decoupling "synth" and "deploy". the main
motivation is to allow "deploy" to run in a controlled environment without needing to execute the app for security purposes.

cdk synth now produces a self-contained assembly in the output directory, which we call a "cloud assembly". this directory includes synthesized templates (similar to current behavior) but also a "manifest.json" file and the asset staging directories.

cdk deploy -a assembly-dir will now skip synthesis and will directly deploy the assembly.

To that end, we modified the behavior of asset staging such that all synth output always goes to the output directory, which is by default cdk.out (can be set with --output). if there's a single template, it is printed to stdout, otherwise the name of output directory is printed.

This PR also includes multiple clean ups and deprecations of stale APIs and modules such as:

cdk.AppProps includes various new options.
An error is thrown if references between stacks that are not in the same app (mostly in test cases).
Added the token creation stack trace for errors emitted by tokens
Added ConstructNode.root which returns the tree root.

TESTING: verified that all integration tests passed and added a few tests to verify zip and docker assets as well as cloud assemblies. See: https://github.com/awslabs/cdk-ops/pull/364

Closes #1893
Closes #2093
Closes #1954
Closes #2310
Related #2073
Closes #1245
Closes #341
Closes #956
Closes #233

BREAKING CHANGE: IMPORTANT: apps created with the CDK version 0.33.0 and above cannot be used with an older CLI version.

--interactive has been removed
--numbered has been removed
--staging is now a boolean flag that indicates whether assets should be copied to the --output directory or directly referenced (--no-staging is useful for e.g. local debugging with SAM CLI)
Assets (e.g. Lambda code assets) are now referenced relative to the output directory.
SynthUtils.templateForStackName has been removed (use SynthUtils.synthesize(stack).template).
cxapi.SynthesizedStack renamed to cxapi.CloudFormationStackArtifact with multiple API changes.
cdk.App.run() now returns a cxapi.CloudAssembly instead of cdk.ISynthesisSession.
cdk.App.synthesizeStack and synthesizeStacks has been removed. The cxapi.CloudAssembly object returned from app.run() can be used as an alternative to explore a synthesized assembly programmatically (resolves App.synthesize methods are marked as deprecated with no replacement #2016).
cdk.CfnElement.creationTimeStamp may now return undefined if there is no stack trace captured.
cdk.ISynthesizable.synthesize now accepts a cxapi.CloudAssemblyBuilder instead of ISynthesisSession.
cdk: The concepts of a synthesis session and session stores have been removed (cdk.FileSystemStore, cdk.InMemoryStore, cdk.SynthesisSession, cdk.ISynthesisSessionandcdk.SynthesisOptions`).
No more support for legacy manifests (cdk.ManifestOptions member legacyManifest has been removed)
All support for build/bundle manifest have been removed (cxapi.BuildManifest, cxapi.BuildStep, etc).
Multiple deprecations and breaking changes in the cxapi module (cxapi.AppRuntime has been renamed to cxapi.RuntimeInfo, cxapi.SynthesizeResponse, cxapi.SynthesizedStack has been removed)
The @aws-cdk/applet-js program is no longer available. Use decdk as an alternative.

verify that old cli fails with a compatibility error and vice versa
run all cli integration tests

Pull Request Checklist

Testing
- Unit test added (prefer not to modify an existing test, otherwise, it's probably a breaking change)
- CLI change?: coordinate update of integration tests with team
- cdk-init template change?: coordinated update of integration tests with team
Docs
- jsdocs: All public APIs documented
- README: README and/or documentation topic updated
- Design: For significant features, design document added to design folder
Title and Description
- Change type: title prefixed with fix, feat and module name in parens, which will appear in changelog
- Title: use lower-case and doesn't end with a period
- Breaking?: last paragraph: "BREAKING CHANGE: <describe what changed + link for details>"
- Issues: Indicate issues fixed via: "Fixes #xxx" or "Closes #xxx"
Sensitive Modules (requires 2 PR approvers)
- IAM Policy Document (in @aws-cdk/aws-iam)
- EC2 Security Groups and ACLs (in @aws-cdk/aws-ec2)
- Grant APIs (only if not based on official documentation with a reference)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license.

formalize the concept of a cloud assembly to allow decoupling "synth" and "deploy". the main motivation is to allow "deploy" to run in a controlled environment without needing to execute the app for security purposes. "cdk synth" now produces a self-contained assembly in the output directory, which we call a "cloud assembly". this directory includes synthesized templates (similar to current behavior) but also a "manifest.json" file and the asset staging directories. "cdk deploy -a <assembly-dir>" will now skip synthesis and will directly deploy the assembly. to that end, we modified the behavior of asset staging such that all synth output always goes to the output directory, which is by default `cdk.out` (can be set with `--output`). if there's a single template, it is printed to stdout, otherwise the name of output directory is printed. BREAKING CHANGE: apps created with this version cannot be deployed using a previous version of the CLI and vice versa. Both have to be >= 0.32.0. - @aws-cdk/cdk.App member synthesizeStack has been removed - @aws-cdk/cdk.App member synthesizeStacks has been removed - @aws-cdk/cdk.FileSystemStore has been removed - @aws-cdk/cdk.InMemoryStore has been removed - @aws-cdk/cdk.SynthesisSession member addBuildStep has been removed - @aws-cdk/cdk.SynthesisSession member store has been removed - @aws-cdk/cdk.FileSystemStoreOptions has been removed - @aws-cdk/cdk.ISessionStore has been removed - @aws-cdk/cdk.ISynthesisSession member addBuildStep has been removed - @aws-cdk/cdk.ISynthesisSession member store has been removed - @aws-cdk/cdk.ManifestOptions member legacyManifest has been removed - @aws-cdk/cdk.ResolveContext has been removed - @aws-cdk/cdk.SynthesisOptions member legacyManifest has been removed - @aws-cdk/cdk.SynthesisOptions member store has been removed - @aws-cdk/cx-api.BuildManifest has been removed - @aws-cdk/cx-api.BuildStep has been removed - @aws-cdk/cx-api.SynthesizeResponse has been removed - @aws-cdk/cx-api.SynthesizedStack has been removed - @aws-cdk/cx-api.BuildStepType has been removed - --interactive has been removed - --numbered has been removed

…mbly

rix0rrr · 2019-05-27T13:35:10Z

if there's a single template, it is printed to stdout, otherwise the name of output directory is printed.

I don't like this behavior. It's good for humans but horrible to script around. Let's make our tools easy to automate, please?

the main
motivation is to allow "deploy" to run in a controlled environment without needing to execute the app for security purposes.

I agree with the ultimate refactorings, which we've talked about previously, but I hope this is not a stepping stone towards cdk deploy in a pipeline. I'd still rather we build a pipeline that deploys through CloudFormation Actions, if possible.

RomainMuller · 2019-05-27T14:01:39Z

if there's a single template, it is printed to stdout, otherwise the name of output directory is printed.

I agree with @rix0rrr on this one. I prefer the tool behaves in exactly one way. If you want to keep the behavior of outputting to stdout for a single template, introduce a CLI flag for it.

packages/@aws-cdk/aws-codecommit/test/test.codecommit.ts

packages/@aws-cdk/cdk/lib/synthesis.ts

packages/@aws-cdk/cx-api/lib/cxapi.ts

packages/aws-cdk/lib/api/cxapp/exec.ts

packages/aws-cdk/lib/api/cxapp/stacks.ts

RomainMuller

That's pretty good overall! I don't have much to say here, besides a couple of things that look a little odd to me (possibly waved away with an explanation?) or having names that I don't like much.

packages/@aws-cdk/applet-js/package.json

packages/@aws-cdk/assert/lib/synth-utils.ts

packages/@aws-cdk/assert/test/test.assertions.ts

packages/@aws-cdk/cdk/lib/app.ts

RomainMuller · 2019-05-27T14:39:47Z

packages/@aws-cdk/cx-api/lib/assembly/cloud-assembly.ts

+
+  private validateDeps() {
+    for (const artifact of this.artifacts) {
+      ignore(artifact.depends);


Should this read artifact.dependencies?

packages/@aws-cdk/cx-api/lib/assembly/cloud-assembly.ts

RomainMuller · 2019-05-27T14:43:52Z

packages/@aws-cdk/cx-api/lib/assembly/cloudformation-stack.ts

+    this.assets = this.buildAssets();
+  }
+
+  private buildAssets() {


This sort of suggested to me that the actual build process was happening here - but this really just makes an array... Could call collectAssets instead.

packages/aws-cdk/bin/cdk.ts

RomainMuller · 2019-05-27T14:52:19Z

packages/aws-cdk/lib/commands/context.ts

@@ -88,7 +88,7 @@ function invalidateContext(context: Context, key: string) {
  // Unset!
  if (context.has(key)) {
    context.unset(key);
-    print(`Context value ${colors.blue(key)} reset. It will be refreshed on the next SDK synthesis run.`);
+    print(`Context value ${colors.blue(key)} reset. It will be refreshed on next synthesis`);


Maybe this should hint that it means network + credentials are needed on the next execution.

- return CloudAssembly from app.run() - change cloud assembly version to 0.33.0 - make stack trace optional in metadata

mergify · 2019-09-23T15:06:11Z

Thanks so much for taking the time to contribute to the AWS CDK ❤️

We will shortly assign someone to review this pull request and help get it
merged. In the meantime, please take a minute to make sure you follow this
checklist:

PR title type(scope): text
- type: fix, feat, refactor go into CHANGELOG, chore is hidden
- scope: name of module without aws- or cdk- prefix or postfix (e.g. s3 instead of aws-s3-deployment)
- text: use all lower-case, do not end with a period, do not include issue refs
PR Description
- Rationale: describe rationale of change and approach taken
- Issues: indicate issues fixed via: fixes #xxx or closes #xxx
- Breaking?: last paragraph: BREAKING CHANGE: <describe what changed + link for details>
Testing
- Unit test added. Prefer to add a new test rather than modify existing tests
- CLI or init templates change? Re-run/add CLI integration tests
Documentation
- README: update module README to describe new features
- API docs: public APIs must be documented. Copy from official AWS docs when possible
- Design: for significant features, follow design process

Elad Ben-Israel added 2 commits May 23, 2019 22:27

interim

b5641a3

eladb requested a review from a team as a code owner May 24, 2019 21:04

eladb mentioned this pull request May 24, 2019

feat(app-delivery): continuous delivery for CDK apps #2073

Closed

4 tasks

Elad Ben-Israel added 6 commits May 25, 2019 00:06

Merge remote-tracking branch 'origin/master' into benisrae/cloud-asse…

2aac0f9

…mbly

deprecate applets

a29f289

a few test fixes

76d4abc

update more tests

f49ef1b

fix lambda test

27323a5

fix codecommit test

09a7b2f

eladb requested review from RomainMuller and skinny85 as code owners May 24, 2019 22:09

Elad Ben-Israel added 6 commits May 25, 2019 01:39

fix a few more tests

2c6e184

fix decdk test

e004555

add missing package.lock

d4fb3c0

add semver back

7d42d2f

cli: fail if unable to resolve aws region

d0f6ae0

cli: support relative asset paths

4f7e680

rix0rrr approved these changes May 27, 2019

View reviewed changes

RomainMuller reviewed May 27, 2019

View reviewed changes

Elad Ben-Israel added 8 commits May 28, 2019 12:40

add "deprecated" to applet-js package.json

2962c9b

update error message when --output is undefined

6cc76d1

rename variable in test.assertions

d3d3e11

move jest config to package.json

e2cbec1

misc

ee39a2e

- return CloudAssembly from app.run() - change cloud assembly version to 0.33.0 - make stack trace optional in metadata

add support for assets in testStack

8f67b13

save & emit token creation stack traces

a4c776a

app.isApp

9fe08a6

eladb deleted the benisrae/cloud-assembly branch May 29, 2019 18:41

piradeepk mentioned this pull request Jun 6, 2019

fix(aws-ecs-patterns): remove LoadBalancedFargateServiceApplet as applets are no longer supported #2779

Merged

4 tasks

robertd mentioned this pull request Jul 8, 2019

cdk synth generates only json files when used with -o #2965

Closed

NGL321 added the contribution/core This is a PR that came from AWS. label Sep 23, 2019

This was referenced Jan 20, 2020

[Snyk] Upgrade @aws-cdk/aws-route53 from 0.22.0 to 0.39.0 MechanicalRock/tech-radar#20

Closed

[Snyk] Upgrade @aws-cdk/aws-s3 from 0.22.0 to 0.39.0 MechanicalRock/tech-radar#21

Closed

lyonzy mentioned this pull request Aug 31, 2021

Allow setting stackName of Stack inside the Stack #7784

Closed

rix0rrr mentioned this pull request Sep 27, 2021

cli: synth fails if cdk.out does not exist #3717

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: decouple "synth" and "deploy" through cloud assemblies #2636

feat: decouple "synth" and "deploy" through cloud assemblies #2636

eladb commented May 24, 2019 •

edited

rix0rrr commented May 27, 2019

RomainMuller commented May 27, 2019

RomainMuller left a comment

RomainMuller May 27, 2019

RomainMuller May 27, 2019

RomainMuller May 27, 2019

mergify bot commented Sep 23, 2019

feat: decouple "synth" and "deploy" through cloud assemblies #2636

feat: decouple "synth" and "deploy" through cloud assemblies #2636

Conversation

eladb commented May 24, 2019 • edited

Pull Request Checklist

rix0rrr commented May 27, 2019

RomainMuller commented May 27, 2019

RomainMuller left a comment

Choose a reason for hiding this comment

RomainMuller May 27, 2019

Choose a reason for hiding this comment

RomainMuller May 27, 2019

Choose a reason for hiding this comment

RomainMuller May 27, 2019

Choose a reason for hiding this comment

mergify bot commented Sep 23, 2019

eladb commented May 24, 2019 •

edited