CDK Bootstrap should not try to process cdk.json

[asnaseer-resilient]

I noticed that running cdk bootstrap causes it to process the cdk.json file (if one is present). I believe this is a bug as the bootstrap process should have no reason to look at any specific app configuration.

Reproduction Steps

We use Typescript and we compile this only when deploying our stack (i.e. when running cdk deploy). This means the transpiled Typescript files are not present when running cdk bootstrap as follows:

cdk bootstrap aws://************/eu-west-2

Error Log

Bootstrapping aws://************/eu-west-2
internal/modules/cjs/loader.js:638
    throw err;
    ^

Error: Cannot find module '.build/stacks/number-management-api-stacks.js'

Environment

• CLI Version :1.39.0
• OS :Mac 10.15.4 (19E287)
• Language :Typescript

Snippet of relevant packages in package.json:

"devDependencies": {
    "@aws-cdk/assert": "^1.39.0",
    "@aws-cdk/aws-appsync": "^1.39.0",
    "@aws-cdk/aws-cognito": "^1.39.0",
    "@aws-cdk/aws-ec2": "^1.39.0",
    "@aws-cdk/aws-iam": "^1.39.0",
    "@aws-cdk/aws-lambda": "^1.39.0",
    "@aws-cdk/aws-rds": "^1.39.0",
    "@aws-cdk/aws-s3": "^1.39.0",
    "@aws-cdk/aws-secretsmanager": "^1.39.0",
    "@aws-cdk/aws-sns": "^1.39.0",
    "@aws-cdk/aws-sns-subscriptions": "^1.39.0",
    "@aws-cdk/aws-sqs": "^1.39.0",
    "@aws-cdk/core": "^1.39.0",
    "@types/aws-lambda": "^8.10.51",
    "@types/node": "^14.0.1",
    "aws-cdk": "^1.39.0",
    "aws-lambda": "^1.0.6",
    "aws-sdk": "^2.678.0",
    "ts-node": "^8.10.1",
    "typescript": "^3.8.3"
  },

---

For reference, this is the contents of the cdk.json file:

{
  "app": "node .build/stacks/number-management-api-stacks.js"
}

This is 🐛 Bug Report

[shivlaks]

@asnaseer-resilient hey, this is intended behaviour. cdk bootstrap will check if there is an --app argument in the cdk.json file. The reason for that is because it will then select environments specified in the stacks for that app (if any) for bootstrapping.

The bootstrap command can be invoked outside of a project and this check will not be performed. This means that the environment specified in the command itself will be used (i.e. cdk bootstrap aws://123456789012/us-east-1).

I'm dropping the bug label and marking this as guidance. Let me know if you have any other questions!

[asnaseer-resilient]

I understand your reasoning but it still doesn't /feel/ right to me. The less surprising thing would be for bootstrap to be completely independent of any app-specific settings.

In my case I have had to put in a workaround where I deleted the cdk.json file and instead added the --app flag to all other cdk commands I am running - which is a pain.

[shivlaks]

what's the ideal workflow you would be looking for? we wouldn't be able to drop the current behaviour as there are users that rely on it and it's something that would be breaking backwards compatibility.

Is it desirable to have a flag maybe to opt out of inferring the environment?
I'm not sure about where in the workflow you're bootstrapping, but would love to learn more about that as well.

Aside: out of curiosity, why not use the default settings that new cdk projects ship with. that should just work

i.e.

{
  "app": "npx ts-node bin/new cdk project.ts"
}

[asnaseer-resilient]

A flag to opt out of this behaviour would indeed be useful. We do not use the structure generated by the CLI as it doesn't quite fit in with our way of working. I will try and write up a short summary of how we are using the CDK and add it in a comment here later this evening.

[asnaseer-resilient]

This is how we would have liked to have our project files as:

.
│
├── package.json
├── tsconfig.json
├── cdk.json
├── cdk.context.json
│
├── .build
│   ├── app
│   │   └── ...
│   └── stacks
│       └── ...
│
└── src
    ├── app
    │   └── ...
    └── stacks
        ├── my-stack-a
        │   ├── my-stack-a-construct.ts
        │   └── my-stack-a-stack.ts
        ├── my-stack-b
        │   ├── my-stack-b-construct.ts
        │   └── my-stack-b-stack.ts
        ├── ...
        │
        └── my-stacks.ts

Relevent snippet from package.json:

  ...,
  "scripts": {
    ...,
    "bootstrap-cdk": "cross-env echo Bootstrapping ${AWS_ENVIRONMENT} && cdk bootstrap ${AWS_ENVIRONMENT} -o .build/cdk.out",
    "synth-stack": "cross-env cdk synth ${STACK} -o .build/cdk.out/${STACK} --no-staging --require-approval never",
    "deploy-stack": "cross-env cdk deploy ${STACK} -o .build/cdk.out/${STACK} --no-staging --require-approval never",
    "undeploy-stack": "cross-env cdk destroy ${STACK} -o .build/cdk.out/${STACK}",
    "diff-stack": "cross-env cdk diff ${STACK} -o .build/cdk.out/${STACK}",
    "list-stacks": "cross-env cdk list -o .build/cdk.out/${STACK}",
    ...
  },

Contents of cdk.json:

{
  "app": "node .build/stacks/my-stacks.js"
}

Initial contents of cdk-context.json (this gets populated by more entries during cdk synth process):

{
  "@aws-cdk/core:enableStackNameDuplicates": "true",
  "aws-cdk:enableDiffNoFail": "true"
}

The .build folder is where we place all build output and all transpiled Typescript code.

The src/app folder contains all the Typescript source code for our application.

The src/stacks folder contains all the Typescript source code for our CDK stacks. Each CDK stack consists of a stack file that instantiates one or more constructs.

Our deployment is performed using CircleCI which calls various scripts within our package.json file. It always runs the bootstrap-cdk script first and then proceeds to compile the code and then call the deploy-stack script.

The scripts synth-stack, undeploy-stack, diff-stack, and list-stacks are only run locally as and when needed.

The issue we had was that the bootstrap-cdk script was failing because it was trying to process the cdk.json file that references .build/stacks/my-stacks.js. This file would not exist at this point as we haven'tcompiled any code yet.

We were therefore forced to delete the cdk.json file and update our package.json scripts by adding in the --app parameter as follows:

  ...,
  "scripts": {
    ...,
    "bootstrap-cdk": "cross-env echo Bootstrapping ${AWS_ENVIRONMENT} && cdk bootstrap ${AWS_ENVIRONMENT} -o .build/cdk.out",
    "synth-stack": "cross-env cdk synth ${STACK} --app .build/stacks/my-stacks.js -o .build/cdk.out/${STACK} --no-staging --require-approval never",
    "deploy-stack": "cross-env cdk deploy ${STACK} --app .build/stacks/my-stacks.js -o .build/cdk.out/${STACK} --no-staging --require-approval never",
    "undeploy-stack": "cross-env cdk destroy ${STACK} --app .build/stacks/my-stacks.js -o .build/cdk.out/${STACK}",
    "diff-stack": "cross-env cdk diff ${STACK} --app .build/stacks/my-stacks.js -o .build/cdk.out/${STACK}",
    "list-stacks": "cross-env cdk list --app .build/stacks/my-stacks.js -o .build/cdk.out/${STACK}",
    ...
  },

All of our applications follow the above logic. We felt it was safe for all the applications to call bootstrap-cdk as we assumed this is an idempotent operation.

[github-actions[bot]]

This issue has not received any attention in 1 year. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[rob3c]

This seems like a demoware-level feature rather than something to take seriously. If you're going to include such 'magic', please give a way to disable it.

[eliasbrange]

I managed to disable the automagic inferring of environments (and synthesizing) by specifying an empty app, like this:

cdk bootstrap aws://account/region --app ''