[cli] bootstrap upgrade to version 4 fails after upgrade from 1.60.0 to 1.61.0

[jacques-]

Bootstrapped using new style CDK bootstrap (BootstrapVersion: 3), then upgraded to CDK 1.61. Got errors like:

❌  InitiativeConfigStack-development failed: Error: InitiativeConfigStack-development: publishing assets requires bootstrap stack version '4', found '3'. Please run 'cdk bootstrap' with a newer CLI version.

Did as instructed in the error and #8678 , but got the following:

env CDK_NEW_BOOTSTRAP=1 npx cdk bootstrap --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess aws://123456789012/eu-west-1
(node:69834) ExperimentalWarning: The fs.promises API is experimental
CDK_NEW_BOOTSTRAP set, using new-style bootstrapping
 ⏳  Bootstrapping environment aws://123456789012/eu-west-1...
 ❌  Environment aws://123456789012/eu-west-1 failed bootstrapping: Error: Not downgrading existing bootstrap stack from version '3' to version '0'. Use --force to force.
Not downgrading existing bootstrap stack from version '3' to version '0'. Use --force to force.

Am I missing something obvious here?

• CLI Version : 1.61.0
• Framework Version: 1.61.0
• Node.js Version: v10.16.3
• OS : mac
• Language (Version):

This is 🐛 Bug Report

[tvanhens]

Dug into this a little bit and it looks like the parameter it is looking for is incorrect:

aws-cdk/packages/aws-cdk/lib/api/bootstrap/deploy-bootstrap.ts

Line 64 in 960ef12

template.Resources?.[BOOTSTRAP_VERSION_OUTPUT]?.Properties?.Value,

I believe this should reference BOOTSTRAP_VERSION_RESOURCE instead:

aws-cdk/packages/aws-cdk/lib/api/bootstrap/bootstrap-props.ts

Line 12 in 960ef12

export const BOOTSTRAP_VERSION_RESOURCE = 'CdkBootstrapVersion';

That is assuming it is supposed to be referencing this resource from the bootstrap-template:

aws-cdk/packages/aws-cdk/lib/api/bootstrap/bootstrap-template.yaml

Lines 358 to 364 in 960ef12

	CdkBootstrapVersion:
	Type: AWS::SSM::Parameter
	Properties:
	Type: String
	Name:
	Fn::Sub: '/aws-cdk-bootstrap/${Qualifier}/version'
	Value: 4

[polothy]

Reproduced by doing the following:

1. Bootstrapping a new region with CDK 1.60.0
2. Upgraded to 1.61.0
3. Re-run the same bootstrap command

Got the same error as reported in the ticket. I re-reran the bootstrap command with the --force flag and that appeared to upgrade it (saw the new SSM parameter and KMS alias).

[polothy]

It would be good to hear from a project owner if --force is an OK workaround and really the only problem is that the upgrade version checking logic just isn't working correctly, so the bootstrap upgrade is working as intended.

[shivlaks]

looking into it:

minimal repro steps:

Deploy bootstrap version 3

npm install -g aws-cdk@1.60.0
cdk bootstrap --context @aws-cdk/core:newStyleStackSynthesis=1

Upgrade & try to deploy bootstrap version 4

npm install -g aws-cdk@1.61.0
cdk bootstrap --context @aws-cdk/core:newStyleStackSynthesis=1

will result in:

Environment aws://123456789012/us-east-1 failed bootstrapping: Error: Not downgrading existing bootstrap stack from version '3' to version '0'. Use --force to force.
    at Object.deployBootstrapStack (/usr/local/lib/node_modules/aws-cdk/lib/api/bootstrap/deploy-bootstrap.ts:29:11)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at /usr/local/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:360:24
    at async Promise.all (index 0)
    at CdkToolkit.bootstrap (/usr/local/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:357:5)
    at main (/usr/local/lib/node_modules/aws-cdk/bin/cdk.ts:244:16)
    at initCommandLine (/usr/local/lib/node_modules/aws-cdk/bin/cdk.ts:188:9)
Not downgrading existing bootstrap stack from version '3' to version '0'. Use --force to force.

[shivlaks]

patch to fix bootstrapping issue was released in 1.61.1

[polothy]

Tested and works just fine! Thanks for fix and also releasing so quickly, very much appreciated! 💯

[asterikx]

@shivlaks this is still broken for me. Upgrading from 1.60.0 straight to 1.61.1 fails with the same error message

$ cdk deploy PipelineStack --region eu-central-1
PipelineStack: deploying...

 ❌  PipelineStack failed: Error: PipelineStack: publishing assets requires bootstrap stack version '4', found '3'. Please run 'cdk bootstrap' with a newer CLI version.

CLI and framework version is 1.61.1.

EDIT: could fix it by manually re-running cdk bootstrap in all affected environments as prompted by the error message, but I suspect this shouldn't be required since I was already using the "new" bootstrap style?

[shivlaks]

@asterikx - That's an expected error message and remediation steps for users of the new bootstrap style when upgrading as we've bumped bootstrap version (for new bootstrap users) with some updates to the template.

The issue that required patching was the inability to upgrade the bootstrap stack successfully.

[dhirajkhodade]

• I am using latest version of cdk i.e. 1.64.1
• My cdk.json has this line in context "@aws-cdk/core:newStyleStackSynthesis": true
• I have bootstrapped my target environment properly with admin access policy
• cdk synth command also succeeded

but when I deploy my stack I am getting below validation error

message: 'Parameters [/cdk-bootstrap/hnb659fds/version] referenced by template have types not supported by CloudFormation.',
code: 'ValidationError',
time: 2020-09-27T14:29:39.972Z,
requestId: '80a35b87-36dc-4bb1-b689-3157d9f171a9',
statusCode: 400,
retryable: false,
retryDelay: 278.94358746834325

I am doubtful about below section in my cdk synth command output this section was not there before when I was using aws cdk version 1.53.0

Parameters:
BootstrapVersion:
Type: AWS::SSM::Parameter::Value
Default: /cdk-bootstrap/hnb659fds/version
Description: Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store.
Rules:
CheckBootstrapVersion:
Assertions:
- Assert:
Fn::Not:
- Fn::Contains:
- - "1"
- "2"
- "3"
- Ref: BootstrapVersion
AssertDescription: CDK bootstrap stack version 4 required. Please run 'cdk bootstrap' with a recent version of the CDK CLI

[suyashdb]

Hi @dhirajkhodade @shivlaks @rix0rrr
were you able to find out why you were getting the above error and what resolved it ..?
I am also getting same error.

I am using cdk 1.81.0 and when i synth my code i am getting same assertion error.

Parameters:
  BootstrapVersion:
    Type: AWS::SSM::Parameter::Value<String>
    Default: /cdk-bootstrap/fcc-tsi/version
    Description: Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store.
Rules:
  CheckBootstrapVersion:
    Assertions:
      - Assert:
          Fn::Not:
            - Fn::Contains:
                - - "1"
                  - "2"
                  - "3"
                - Ref: BootstrapVersion
        AssertDescription: CDK bootstrap stack version 4 required. Please run 'cdk bootstrap' with a recent version of the CDK CLI.

MD342PML7J:aws-workload-teradata suya8824$ cdk --version
1.81.0 (build 6ef67c7)

[dhirajkhodade]

Hi @suyashdb yes I have resolved this issue.
There are few scenarios.
First of all I was using old version of CDK which uses old bootstrap version and when I upgraded my CDK app to new version i.e. 1.64.1 which uses new version of bootstrap i.e. bootstrap V4 I started getting above mentioned error.

1. So I had to bootstrap my target AWS account again with new bootstrap version so that it will create required bootstrap parameter in SSM param store
   So you should have this parameter in your param store - "/cdk-bootstrap/${Qualifier}/version"
   where ${Qualifier} is hnb659fds but looking at your error I think you are not using default qualifier, you are using
   qualifier - fcc-tsi
   So you should have this parameter in you param store /cdk-bootstrap/fcc-tsi/version

2. Another thing is your parameter type should be string and not SecureString because aws cloud formation template does not support SecureString yet.
   Our cloud security team converted all the parameters in our param store to SecureString and that took some time for me to realize.