feat(cli): preview of `cdk import` #17666

tomas-mazak · 2021-11-23T18:45:37Z

An initial version of cdk import, bringing existing resources under the
management of CloudFormation.

To use:

Make sure your diff is clean (you've recently deployed)
Add constructs for the resource(s) you want to import. Make sure the CDK code configures them exactly as they are configured in reality.
- You can provide resource names here but it's probably better if you don't.
Run cdk import. If you use v2, this requires updating the bootstrap stack.
Provide the actual resource names for each resource (if necessary).
An importing changeset will execute and the resources are imported.

This is an implementation of aws/aws-cdk-rfcs#52

gitpod-io · 2021-11-23T19:12:41Z

rix0rrr

❤️ love it! Thanks for working on this!

rix0rrr · 2021-11-24T12:56:01Z

packages/aws-cdk/README.md

+  **no other changes must be done to the stack before the import is completed**
+- run `cdk deploy` with `--import-resources` argument to instruct CDK to start the import


Feels like this is a different enough workflow that it's worth its own command. How about cdk import-resources (cdk import for short) ?

Probably the -e flag should be implied?

sure, I didn't spend too much thought on the interface TBH, I considered this more like a PoC, to try out how it would work ... I am happy to raise a RFC - actually I only read about the RFC process after raising this PR and so I started with it right away by raising it at #aws-cdk-rfcs Slack channel.

Happy to change the interface to your liking.

updated accordingly

rix0rrr · 2021-11-24T12:59:31Z

packages/aws-cdk/lib/import.ts

+        identifier[idpart] = chg.newProperties[idpart];
+      } else {
+        const displayName : string = newTemplate.template?.Resources?.[id]?.Metadata?.['aws:cdk:path'] ?? id;
+        identifier[idpart] = await promptly.prompt(`Please enter ${idpart} of ${chg.newResourceType} to import as ${displayName.replace(/\/Resource$/, '')}: `);


It might be helpful to be able to read these answers from a file as well (JSON file?) so people can use it in an unattended way.

rix0rrr · 2021-11-24T13:00:36Z

packages/aws-cdk/lib/import.ts

+      }
+    }
+
+    resourcesToImport.push({


For the interactive flow (which I super love!) I think it's probably worth printing a summary with an option to confirm (or re-enter the names), wdyt?

rix0rrr · 2021-11-24T13:02:39Z

packages/aws-cdk/lib/import.ts

+    resourcesToImport.push({
+      LogicalResourceId: id,
+      ResourceType: chg.newResourceType,
+      ResourceIdentifier: identifier,


Oh god must we really always supply this, even if it's already in the template? Great job on automating that away!

Yeah, the CFN API for importing is suboptimal ... :/ This structure can at least be automatically assembled, but what's worse, a changeset of type import cannot contain any non-import template changes apparently. It will be a challenge to support an import of higher-level constructs that synthesise into multiple CFN resources of which only a subset must be imported and the rest needs to be created.

It will be a challenge to support an import of higher-level constructs that synthesise into multiple CFN resources of which only a subset must be imported and the rest needs to be created.

That is a good point. What we could be doing is:

From all the resources that are added, have the user pick which ones they want to import (exact UI for this TBD)

Cull the template to only selected resources

Run an import with only those resources

(Potentially: follow up with a regular deploy)

By necessity the resources that are imported cannot depend on other resources that aren't being imported as well (or exist already) so that's a bit limiting but best we can do for now until we build a more comprehensive multi-phase importing engine?

That sounds like a great idea, I will try to implement that!

As for the UI, it might be as simple as:

For resources we don't know a name for, if the user doesn't fill in a name we skip it.

For sources we DO know a name for, we pre-fill it and the user can backspace it if they don't want to import.

The latter may not be possible with promptly, I'm not quite sure. But that would seem simple and clear enough.

@rix0rrr , I have implemented the above flow: pre-filling a prompt wasn't possible (or I didn't find out how) but it now works that:

if the identifier can be inferred from props, only ask whether to import the given resource or skip

if the identifier can't be inferred, ask for the identifier interactively, if left empty, skip import

All resources that don't support importing or were marked to be skipped by user will be removed from the template for the import operation. So the import succeeds and the subsequent deploy will create the remaining resources.

What do you think?

That sounds great!

rix0rrr · 2021-11-24T13:04:06Z

Haha most of the things I was asking for were already in your TODO list 😅

…uckets only) This is an initial proposal to support existing resources import into CDK stacks. As a PoC, this PR shows a working solution for S3 buckets. This is achieved by introducing `-i` / `--import-resources` CLI option to the `cdk deploy` command. If specified, the newly added resources will not be created, but attempted to be imported (adopted) instead. If the resource definition contains the full resource identifier, this happens automatically. For resources that can't be identified (e.g. an S3 bucket without an explicit `bucketName`), user will be prompted for the necessary information.

…by CloudFormation

tgigsigniant · 2022-01-10T18:42:48Z

Hi @tomas-mazak, I was just wondering if CDK import will natively support importing Global tables for DynamoDB as part of this PR (AWS Support directed me towards this PR)?

Additional details:
We currently had to hand roll our own solution to import a Global Table to a CDK app by using the CFN import tool. Our solution was based off of this stack overflow post . But, we ran into several errors specific to importing a Global Table in CDK. For example, we couldn't import the Table with the replicationRegions param specified given all the additional AssetParameters it generates.

An error occurred (ValidationError) when calling the CreateChangeSet operation: Parameters: [AssetParametersc691172cdeefa2c91b5a2907f9d81118e47597634943344795f1a844192dd49cS3BucketEAC9DD43, ...] must have value

To get around this, we didn't include replicationRegions for the import, and added it back for the next deployment after the import. However, the replicationRegions param doesn't support tables with existing replicas and requires deleting them first and then recreating them with CDK if we want them to be managed by CDK.

tomas-mazak · 2022-01-12T10:13:37Z

Hi @tomas-mazak, I was just wondering if CDK import will natively support importing Global tables for DynamoDB as part of this PR (AWS Support directed me towards this PR)?

@tgigsigniant , this PR is a (thin) layer on top of the CloudFormation import feature. Instead of doing several manual steps, like in the stackoverflow post you referred to, you will be able to handle the import from CDK command line. All limitations of the CloudFormation import API will still apply. So if CFN doesn't support importing dynamo tables with certain configuration, this PR won't help you unfortunately.

fitzoh · 2022-01-18T03:20:50Z

If/when this gets merged, this portion of the CDK docs should be updated to read "Referencing existing external resources" instead of "Importing existing external resources" You could probably argue that change should happen regardless of this PR/functionality.

henrysachs · 2022-03-07T20:34:46Z

Hey There, really interested in this feature to faster recover rds instances. Any way to support or push this? :)

RichiCoder1 · 2022-03-07T20:46:29Z

Also would love to see this (and even assist if necessary) as this would make Serverless and TF migrations to CDK much simpler.

saltman424 · 2022-03-14T17:54:07Z

Once this is implemented, what would be the recommended way to utilize it as part of @aws-cdk/pipelines or would that require a separate PR?

rix0rrr · 2022-03-23T13:05:15Z

@tomas-mazak I'm doing some work on this PR. No need to jump in.

rix0rrr · 2022-04-05T10:05:34Z

TODO

Informational message at the end about needing to do a drift detection, or cdk deploy if there were ignored non-addition changes

packages/aws-cdk/lib/cli.ts

packages/aws-cdk/lib/import.ts

RomainMuller · 2022-04-05T10:15:44Z

packages/aws-cdk/lib/import.ts

+function fmtdict<A>(xs: Record<string, A>) {
+  return Object.entries(xs).map(([k, v]) => `${k}=${v}`).join(', ');
+}


(Optional/Opportunistic) Any reason not to use JSON.stringify here? Or YAML.stringify?

Mm. Those don't format things the same. I want it human readable.

RomainMuller · 2022-04-05T10:18:17Z

packages/aws-cdk/test/integ/helpers/cdk.ts

-  await shell(['npm', 'install',
-    '--prefix', installDir,
-    'npm@7']);
+  await shell(['npm', 'install', 'npm@7'], { cwd: installDir });


While we're there...

Suggested change

await shell(['npm', 'install', 'npm@7'], { cwd: installDir });

await shell(['npm', 'install', 'npm@8'], { cwd: installDir });

That's another can of worms I'd rather not get into in this PR.

rix0rrr · 2022-03-23T09:33:18Z

packages/aws-cdk/lib/cdk-toolkit.ts

+
+    highlight(stack.displayName);
+
+    if (!stack.environment) {


The environment might be agnostic, and could still be resolved using current credentials.

rix0rrr · 2022-04-05T12:11:42Z

packages/aws-cdk/lib/import.ts

+function fmtdict<A>(xs: Record<string, A>) {
+  return Object.entries(xs).map(([k, v]) => `${k}=${v}`).join(', ');
+}


Mm. Those don't format things the same. I want it human readable.

rix0rrr · 2022-04-05T12:12:05Z

packages/aws-cdk/test/integ/helpers/cdk.ts

-  await shell(['npm', 'install',
-    '--prefix', installDir,
-    'npm@7']);
+  await shell(['npm', 'install', 'npm@7'], { cwd: installDir });


That's another can of worms I'd rather not get into in this PR.

Pull request has been modified.

mergify · 2022-04-05T13:21:07Z

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

aws-cdk-automation · 2022-04-05T14:06:01Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject89A8053A-LhjRyN9kxr8o
Commit ID: a021ed6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mergify · 2022-04-05T14:06:21Z

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

rix0rrr · 2022-04-07T10:07:26Z

Took a while to get this in, but <3 thanks @tomas-mazak for the hard work!

See [CHANGELOG](https://github.com/aws/aws-cdk/blob/bump/1.152.0/CHANGELOG.md) For convenience, extracted the relevant CHANGELOG entry: ## [1.152.0](v1.151.0...v1.152.0) (2022-04-06) ### Features * **cfnspec:** cloudformation spec v63.0.0 ([#19679](#19679)) ([dba96a9](dba96a9)) * **cfnspec:** cloudformation spec v65.0.0 ([#19745](#19745)) ([796fc64](796fc64)) * **cli:** add --build option ([#19663](#19663)) ([eb9b8e2](eb9b8e2)), closes [#19667](#19667) * **cli:** preview of `cdk import` ([#17666](#17666)) ([4f12209](4f12209)) * **core:** throw error when stack name exceeds max length ([#19725](#19725)) ([1ffd45e](1ffd45e)) * **eks:** add k8s v1.22 ([#19756](#19756)) ([9a518c5](9a518c5)) * **opensearch:** Add latest Opensearch Version 1.2 ([#19749](#19749)) ([a2ac36e](a2ac36e)) * add new integration test runner ([#19754](#19754)) ([1b4d010](1b4d010)) * **eks:** alb-controller v2.4.1 ([#19653](#19653)) ([1ec08df](1ec08df)) * **lambda:** add support for ephemeral storage ([#19552](#19552)) ([f1d9b6a](f1d9b6a)), closes [#19605](#19605) * **s3:** EventBridge bucket notifications ([#18614](#18614)) ([d8e602b](d8e602b)), closes [#18076](#18076) * **synthetics:** new puppeteer 3.5 runtime ([#19673](#19673)) ([ce2b91b](ce2b91b)), closes [#19634](#19634) ### Bug Fixes * **aws_applicationautoscaling:** Add missing members to PredefinedMetric enum ([#18978](#18978)) ([75a6fa7](75a6fa7)), closes [#18969](#18969) * **cli:** apps with many resources scroll resource output offscreen ([#19742](#19742)) ([053d22c](053d22c)), closes [#19160](#19160) * **cli:** support attributes of DynamoDB Tables for hotswapping ([#19620](#19620)) ([2321ece](2321ece)), closes [#19421](#19421) * **cloudwatch:** automatic metric math label cannot be suppressed ([#17639](#17639)) ([7fa3bf2](7fa3bf2)) * **codedeploy:** add name validation for Application, Deployment Group and Deployment Configuration ([#19473](#19473)) ([9185042](9185042)) * **codedeploy:** the Service Principal is wrong in isolated regions ([#19729](#19729)) ([7e9a43d](7e9a43d)), closes [#19399](#19399) * **core:** `Fn.select` incorrectly short-circuits complex expressions ([#19680](#19680)) ([7f26fad](7f26fad)) * **core:** detect and resolve stringified number tokens ([#19578](#19578)) ([7d9ab2a](7d9ab2a)), closes [#19546](#19546) [#19550](#19550) * **core:** reduce CFN template indent size to save bytes ([#19656](#19656)) ([fd63ca3](fd63ca3)) * **ecs:** 'desiredCount' and 'ephemeralStorageGiB' cannot be tokens ([#19453](#19453)) ([c852239](c852239)), closes [#16648](#16648) * **ecs:** remove unnecessary error when adding volume to external task definition ([#19774](#19774)) ([5446ded](5446ded)), closes [#19259](#19259) * **iam:** policies aren't minimized as far as possible ([#19764](#19764)) ([876ed8a](876ed8a)), closes [#19751](#19751) * **logs:** Faulty Resource Policy Generated ([#19640](#19640)) ([1fdf122](1fdf122)), closes [#17544](#17544)

See [CHANGELOG](https://github.com/aws/aws-cdk/blob/bump/2.20.0/CHANGELOG.md) For convenience, extracted the relevant CHANGELOG entry: ## [2.20.0](v2.19.0...v2.20.0) (2022-04-07) ### Features * **cfnspec:** cloudformation spec v63.0.0 ([#19679](#19679)) ([dba96a9](dba96a9)) * **cfnspec:** cloudformation spec v65.0.0 ([#19745](#19745)) ([796fc64](796fc64)) * **cli:** add --build option ([#19663](#19663)) ([eb9b8e2](eb9b8e2)), closes [#19667](#19667) * **cli:** preview of `cdk import` ([#17666](#17666)) ([4f12209](4f12209)) * **core:** throw error when stack name exceeds max length ([#19725](#19725)) ([1ffd45e](1ffd45e)) * **eks:** add k8s v1.22 ([#19756](#19756)) ([9a518c5](9a518c5)) * **opensearch:** Add latest Opensearch Version 1.2 ([#19749](#19749)) ([a2ac36e](a2ac36e)) * add new integration test runner ([#19754](#19754)) ([1b4d010](1b4d010)) * **eks:** alb-controller v2.4.1 ([#19653](#19653)) ([1ec08df](1ec08df)) * **lambda:** add support for ephemeral storage ([#19552](#19552)) ([f1d9b6a](f1d9b6a)), closes [#19605](#19605) * **s3:** EventBridge bucket notifications ([#18614](#18614)) ([d8e602b](d8e602b)), closes [#18076](#18076) ### Bug Fixes * **aws_applicationautoscaling:** Add missing members to PredefinedMetric enum ([#18978](#18978)) ([75a6fa7](75a6fa7)), closes [#18969](#18969) * **cli:** apps with many resources scroll resource output offscreen ([#19742](#19742)) ([053d22c](053d22c)), closes [#19160](#19160) * **cli:** support attributes of DynamoDB Tables for hotswapping ([#19620](#19620)) ([2321ece](2321ece)), closes [#19421](#19421) * **cloudwatch:** automatic metric math label cannot be suppressed ([#17639](#17639)) ([7fa3bf2](7fa3bf2)) * **codedeploy:** add name validation for Application, Deployment Group and Deployment Configuration ([#19473](#19473)) ([9185042](9185042)) * **codedeploy:** the Service Principal is wrong in isolated regions ([#19729](#19729)) ([7e9a43d](7e9a43d)), closes [#19399](#19399) * **core:** `Fn.select` incorrectly short-circuits complex expressions ([#19680](#19680)) ([7f26fad](7f26fad)) * **core:** detect and resolve stringified number tokens ([#19578](#19578)) ([7d9ab2a](7d9ab2a)), closes [#19546](#19546) [#19550](#19550) * **core:** reduce CFN template indent size to save bytes ([#19656](#19656)) ([fd63ca3](fd63ca3)) * **ecs:** 'desiredCount' and 'ephemeralStorageGiB' cannot be tokens ([#19453](#19453)) ([c852239](c852239)), closes [#16648](#16648) * **ecs:** remove unnecessary error when adding volume to external task definition ([#19774](#19774)) ([5446ded](5446ded)), closes [#19259](#19259) * **iam:** policies aren't minimized as far as possible ([#19764](#19764)) ([876ed8a](876ed8a)), closes [#19751](#19751) * **logs:** Faulty Resource Policy Generated ([#19640](#19640)) ([1fdf122](1fdf122)), closes [#17544](#17544)

tomas-mazak · 2022-04-07T11:16:14Z

Thanks a lot @rix0rrr for taking over! Glad it's now in!

mmuller88 · 2022-04-10T01:47:18Z

Seems useful thanks to everyone working on this and to @pahud for the image.

An initial version of `cdk import`, bringing existing resources under the management of CloudFormation. To use: - Make sure your diff is clean (you've recently deployed) - Add constructs for the resource(s) you want to import. **Make sure the CDK code configures them exactly as they are configured in reality**. - You can provide resource names here but it's probably better if you don't. - Run `cdk import` - Provide the actual resource names for each resource (if necessary). - An importing changeset will execute and the resources are imported. This is an implementation of aws/aws-cdk-rfcs#52

github-actions bot added the package/tools Related to AWS CDK Tools or CLI label Nov 23, 2021

github-actions bot assigned rix0rrr Nov 23, 2021

tomas-mazak force-pushed the feat/import_resources branch from e480ab1 to 273569d Compare November 23, 2021 19:12

tomas-mazak force-pushed the feat/import_resources branch from 273569d to 63ce640 Compare November 23, 2021 19:21

rix0rrr reviewed Nov 24, 2021

View reviewed changes

tomas-mazak force-pushed the feat/import_resources branch from 63ce640 to 9e4c131 Compare November 27, 2021 00:39

tomas-mazak changed the title ~~feat(cli): Add an option to import existing resources (currently S3 buckets only)~~ feat(cli): Add an option to import existing resources Nov 27, 2021

tomas-mazak added 3 commits December 13, 2021 16:17

feat(cli): Extend cdk import support to all resource types supported …

c424fce

…by CloudFormation

feat(cli): add support for providing import resource mapping from a file

a06a1cd

tomas-mazak force-pushed the feat/import_resources branch from 9e4c131 to a06a1cd Compare December 13, 2021 15:21

rix0rrr mentioned this pull request Dec 24, 2021

Allow declarative/CLI method to import pre-existing resources #18022

Closed

2 tasks

Merge remote-tracking branch 'origin/master' into pr/tomas-mazak/17666

c79c34f

rix0rrr changed the title ~~feat(cli): Add an option to import existing resources~~ feat(cli): preview of cdk import Mar 23, 2022

Fix type complaints

c43dd1b

rix0rrr added 6 commits March 24, 2022 12:41

Slight refactoring

ed65bfc

Need to use deploy-role after all

5f76731

Add integ test

f995816

Change alias so it doesn't conflict with output

77d0f21

Starting to add some unit tests

29d8b89

Adding more tests

2372f24

rix0rrr added 2 commits March 24, 2022 17:57

Add final unit test

1e8ea73

Merge branch 'master' into feat/import_resources

cc70cfe

RomainMuller reviewed Apr 5, 2022

View reviewed changes

rix0rrr reviewed Apr 5, 2022

View reviewed changes

Improve messaging

6436056

rix0rrr previously approved these changes Apr 5, 2022

View reviewed changes

Fix tests

ad572e8

rix0rrr approved these changes Apr 5, 2022

View reviewed changes

Merge branch 'master' into feat/import_resources

a021ed6

mergify bot merged commit 4f12209 into aws:master Apr 5, 2022

github-actions bot added the p2 label Apr 7, 2022

This was referenced Apr 7, 2022

chore(release): 1.152.0 #19781

Merged

chore(release): 2.20.0 #19801

Merged

amrhesham-fci mentioned this pull request Oct 8, 2023

Support resource import aws/aws-cdk-rfcs#52

Closed

7 tasks

		no other changes must be done to the stack before the import is completed
		- run `cdk deploy` with `--import-resources` argument to instruct CDK to start the import

	await shell(['npm', 'install', 'npm@7'], { cwd: installDir });
	await shell(['npm', 'install', 'npm@8'], { cwd: installDir });

feat(cli): preview of cdk import #17666

feat(cli): preview of cdk import #17666

Conversation

tomas-mazak commented Nov 23, 2021 • edited by rix0rrr

gitpod-io bot commented Nov 23, 2021

rix0rrr left a comment

Choose a reason for hiding this comment

rix0rrr Nov 24, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rix0rrr Nov 24, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rix0rrr Nov 24, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rix0rrr commented Nov 24, 2021

tgigsigniant commented Jan 10, 2022

tomas-mazak commented Jan 12, 2022

fitzoh commented Jan 18, 2022

henrysachs commented Mar 7, 2022

RichiCoder1 commented Mar 7, 2022

saltman424 commented Mar 14, 2022

rix0rrr commented Mar 23, 2022

rix0rrr commented Apr 5, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mergify bot commented Apr 5, 2022

aws-cdk-automation commented Apr 5, 2022

AWS CodeBuild CI Report

mergify bot commented Apr 5, 2022

rix0rrr commented Apr 7, 2022 • edited

tomas-mazak commented Apr 7, 2022

mmuller88 commented Apr 10, 2022

feat(cli): preview of `cdk import` #17666

feat(cli): preview of `cdk import` #17666

tomas-mazak commented Nov 23, 2021 •

edited by rix0rrr

rix0rrr Nov 24, 2021 •

edited

rix0rrr Nov 24, 2021 •

edited

rix0rrr Nov 24, 2021 •

edited

rix0rrr commented Apr 5, 2022 •

edited

rix0rrr commented Apr 7, 2022 •

edited