Extending Copilot with the CDK

[efekarakus]

Hi folks! This issue is a design proposal for overriding Copilot abstracted resources using the AWS Cloud Development Kit (CDK). The goal is to provide a "break the glass" mechanism to access and configure functionality that is not surfaced by Copilot manifests by leveraging the expressive power of a programming language.

Problem statement

Clients can describe their application configuration with declarative manifest.yml files. Manifests enable users to define their application in terms of “architecture as code” that gets transformed into AWS CloudFormation (CFN) templates by Copilot. Today, customers have the ability to define additional AWS resources with “addons” CFN templates. However, the resources encapsulated by manifests remain unmodifiable by users. We're looking into providing extension solutions for customers to self-unblock themselves and access the internal layers of Copilot.

This document proposes a user experience for extending manifests with the CDK.

Proposal

1. Copilot will introduce a new command copilot [noun] override to create a CDK application that overrides Copilot resources, for example:

   $ copilot svc override
   > Which service's resources would you like to override?
   * frontend
   * api
   
   > Which infrastructure as code tool would you like to use to override "frontend"?
   * AWS Cloud Development Kit (CDK)
   * CloudFormation YAML patches
   
   > Which programming language would you like to use with the CDK?
   * Typescript(ts)
   
   > Which resources in "frontend" would you like to to override?
   [ ] DiscoveryService          (AWS::ServiceDiscovery::Service)
   [x] Service                   (AWS::ECS::Service)
   [x] PublicNetworkLoadBalancer  (AWS::ElasticLoadBalancingV2::LoadBalancer)
   [ ] TaskDefinition            (AWS::ECS::TaskDefinition)
   ✅ Created a new CDK app under `copilot/frontend/overrides/` to override the resources:
   .
   ├── .gitignore
   ├── .build/
   ├── bin/
   │   └── override.ts
   ├── cdk.json
   ├── package.json
   ├── stack.ts
   ├── tsconfig.json
   └── README.md
   
   Copilot did not detect npm installed.
   > Would you like Copilot to install npm? [Y/n] y
   ✅ Installed npm v9.1.1 and NodeJS v1.18.2.
   ✅ Ran `npm install` to install dependencies.
   
   > Recommended Actions:
   1. Please follow the guide under README.md

   The file structure generated by the override command is equivalent to cdk init output. The only additional file created by Copilot is the hidden .build/ directory.

   In order to simplify onboarding with the CDK, Copilot will assist clients with installing their dependencies. On macOS and Linux, Copilot can install npm for customers on their local machine. On Windows, we won't be able to help customers with installing npm and will skip to asking them to follow the README instead. Once npm is present, Copilot will run npm install to install the dependencies on behalf of the client.

2. Clients have to edit only stack.ts file in order to implement the transform* methods for the selected resources. The following content will be present in the file by default:

   import * as cdk from 'aws-cdk-lib';
   import { aws_elasticloadbalancingv2 as elbv2 } from 'aws-cdk-lib';
   import { aws_ecs as ecs } from 'aws-cdk-lib';
   
   export class TransformedStack extends cdk.Stack {
       constructor (scope: cdk.App, id: string, props?: cdk.StackProps) {
           super(scope, id, props);
            this.template = new cdk.cloudformation_include.CfnInclude(this, 'Template', {
               templateFile: path.join('.build', 'in.yaml'),
           });
           this.appName = template.getParameter('AppName').valueAsString;
           this.envName = template.getParameter('EnvName').valueAsString;
   
           this.transformService();
           this.transformPublicNetworkLoadBalancer();
       }
    
       // TODO: implement me.
       transformService() {
         const service = this.template.getResource("Service") as ecs.CfnService;
         throw new error("not implemented");
       }
   
       // TODO: implement me.
       transformPublicNetworkLoadBalancer() {
         const publicNetworkLoadBalancer = this.template.getResource("PublicNetworkLoadBalancer") as elbv2.CfnLoadBalancer;
         throw new error("not implemented");
       }
   }

   As can be seen above, Copilot will use the clouformation_include module provided by the CDK to help author transformations. This library is the CDK’s recommendation from their “Import or migrate an existing AWS CloudFormation template” guide. It enables accessing the resources hidden by the manifest as L1 constructs and ensures the CDK retains the resources on synthesis. The CfnInclude object is initialized from the .build/in.yaml CFN template. This is how Copilot and the CDK communicates. Copilot writes the manifest generated CFN template under the .build/ directory, which then gets parsed by the cloudformation_include library into a CDK construct.

That's it! From this point forward, copilot svc package or copilot svc deploy will apply the overrides written in stack.ts. Under the hood, Copilot will invoke cdk synth to synthesize the transformed template and use that to deploy to the existing CFN stack.

Feedback

We'd love to hear your feedback on the user experience above:

1. Do you see yourself using this feature? (overriding resources with the CDK)
2. Does the feature meet your needs?
3. If not, can you elaborate on what you'd like to do that can't be done with the above proposal?

Appendix: Sample issues

Every new “capability” feature request can be boiled down to having an override functionality. I’ve captured only a sample of these to test the proposal against.

Link	Description
#4063	Override nlb configuration to assign elastic IPs to the network load balancer
#4010	Assign ReadOnlyRootFS: true to all containers in the task definition by default
#4005, #3840	Support for internal NLB for Backend Services. Customers can create the nlb with addons/ but need to wire it by overriding the ECS::Service definition
#3733	Allow setting external launch type for ECS tasks
#3721	Allow setting VPC flow logs for environments
#3720	Turn off assigning public IP addresses for tasks launched in public subnets
#3594, #1783	Multiple port support for services behind NLB and ALB
#3506	Add additional permissions to the generated environment manager role
#3504	Specify additional security groups for the app runner vpc connector

[efekarakus]

For an alternative proposal that uses simple CloudFormation YAML patches, see #4209!

[RichiCoder1]

I really love this! The only addition I'd have is to take it even further. A lot of the power of the CDK is high level abstractions, so it'd be nice to have a Copilot abstraction that takes care of details like knowing where/how to template include and possibly having opinionated/type-safe ways of getting templated resources.

[efekarakus]

Hi @RichiCoder1 👋

Thanks for the feedback! Are you thinking of something along these lines:

import * as copilot from 'aws-copilot-lib';

const props = copilot.buildPropsFromManifest('../manifest.yml');
const lbws = new copilot.LoadBalancedWebService(this, 'Web', props);
const service = lbws.getService(); // returns L2 ECS service construct.                  

One benefit that I can think of with this route is that it would allow you to compose CDK apps made out of Copilot abstractions 👍.
Or were you thinking of a different option?

---

that takes care of details like knowing where/how to template include

For clarification, when running copilot svc/env override, Copilot will scaffold the following piece of code:

constructor (scope: cdk.App, id: string, props?: cdk.StackProps) {
      super(scope, id, props);
       this.template = new cdk.cloudformation_include.CfnInclude(this, 'Template', {
          templateFile: path.join('.build', 'in.yaml'), 
      });

And everytime copilot svc package or copilot deploy is run Copilot will write the CloudFormation template automatically under .build/in.yaml so no user action is needed.

opinionated/type-safe ways of getting templated resources

Similarly, Copilot will scaffold the following line:

const service = this.template.getResource("Service") as ecs.CfnService;

The service variable is now a type-safe L1 CDK construct: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ecs.CfnService.html

Do these responses mitigate your concerns?

[benjaminpottier]

I think this is great! I have a few questions/comments:

• Will there be a beta we can use before the final release of this feature (if it is released)?
• Currently, any generated cloudformation from copilot isn't visible until its actually deployed to AWS. Would there be a synth-like command that could be run from copilot to show us everything that will be deployed (not just the CDK we've written to override/expand services/jobs etc.)
• I think having constructs that represents the abstracted "service" or "job" etc would be very helpful. That way we can work with a single construct as opposed to working individually with each resource that composes the abstraction/pattern. See: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ecs_patterns-readme.html ... I'm not sure of the patterns here track with what copilot releases.

[Lou1415926]

Hello @benjaminpottier!

I can probably answer your second question - inviting @efekarakus for the other two!

Currently, any generated cloudformation from copilot isn't visible until its actually deployed to AWS. Would there be a synth-like command that could be run from copilot to show us everything that will be deployed (not just the CDK we've written to override/expand services/jobs etc.)

You should be able to run copilot svc package to preview the CFN template that would be generated after applying the CDK code changes. In addition, we are thinking about adding --diff flag to copilot svc package for folks to view the difference between the deployed template and the local template. Would these be helpful for you?

[bpottier]

Hello @benjaminpottier!

I can probably answer your second question - inviting @efekarakus for the other two!

Currently, any generated cloudformation from copilot isn't visible until its actually deployed to AWS. Would there be a synth-like command that could be run from copilot to show us everything that will be deployed (not just the CDK we've written to override/expand services/jobs etc.)

You should be able to run copilot svc package to preview the CFN template that would be generated after applying the CDK code changes. In addition, we are thinking about adding --diff flag to copilot svc package for folks to view the difference between the deployed template and the local template. Would these be helpful for you?

That answers my question. Thank you! And yes, diff would be helpful.

[craigjbass]

We are currently planning to use copilot in "compliance environments" (UK Public Sector), where individual development teams have limited access to AWS APIs (i.e. no ability to apply CF from local terminal).

What this looks like is using Copilot strategically as a tool as part of a larger "platform offering" from a centralised platform team. I do wonder how these advanced override features will evolve over time and be suitable in those scenarios.

My current view is that there would either need to be an abstraction around copilot, or some sort of copilot-managed concept that groups "Applications" into a wider platform, and allows cross-cutting changes to be applied to many teams copilot managed services.

[efekarakus]

Hi @bpottier, @craigjbass !
Apologies on the late reply, I was at re:invent 😎

Will there be a beta we can use before the final release of this feature (if it is released)?

We can happily share a pre-release binary here before submission to get early feedback on the feature, that's a great idea 👍

---

I think having constructs that represents the abstracted "service" or "job" etc would be very helpful.

That's really interesting, we can definitely create a L3 construct that takes the same manifest file as an object as input. Similar to in this comment:

const lbws = new copilot.LoadBalancedWebService(this, 'Web', {
  http: {
     path: '/',
     targetPort: 80,
  },
  image: {
     build: './Dockerfile',
  },
});

Are you inclined for this abstraction so that you can have all your infrastructure defined in the CDK?

Perhaps that would be a better fit for a copilot eject command

I'm guessing that the Copilot CDK construct would still be interoperable with Copilot commands like copilot svc logs.
Replacing the infrastructure definition from Copilot manifests to the CDK seems to be more around "ejecting" rather than extending Copilot. Which I think is understandable, the aws-copilot-lib library would be used to replace manifests and interoperate with the CDK, while Copilot the CLI can be used for operational commands.

---

My current view is that there would either need to be an abstraction around copilot, or some sort of copilot-managed concept that groups "Applications" into a wider platform, and allows cross-cutting changes to be applied to many teams copilot managed services.

Interesting! Can I ask some clarification questions @craigjbass 🥺:

1. Do you deem it's okay if this feature is reserved to the platform team rather than the dev teams?
2. Out of curiosity: How will dev teams make changes to their services? Do they have access to the manifest.yml files and can do a git push to trigger the copilot pipeline to apply their changes?
   3.Is the concern applying the same "extension" across multiple manifest files? If so I think a pattern like this can be possible:

   import {customTransform} from '../../common/transform-copilot.ts'; // Import from a common path or a published node module.
   
   constructor (scope: cdk.App, id: string, props?: cdk.StackProps) {
      super(scope, id, props);
       this.template = new cdk.cloudformation_include.CfnInclude(this, 'Template', {
          templateFile: path.join('.build', 'in.yaml'), 
      });
      customTransform(this.template.getResource("Service");  // Apply the transformation here.
   }

[craigjbass]

1. Do you deem it's okay if this feature is reserved to the platform team rather than the dev teams?

Yes, although I wonder how we'd enforce that without simply not providing the Copilot CLI to teams. I can see a benefit in giving some but not all of the functionality of the Copilot CLI directly to dev teams. (Mostly around operability)

2. Out of curiosity: How will dev teams make changes to their services? Do they have access to the manifest.yml files and can do a git push to trigger the copilot pipeline to apply their changes?

Unclear yet. Ideally, they would just use the copilot manifest, but if we can't put in the relevant controls - they will probably commit a manifest that a custom tool will read to produce the copilot manifest. Examples of areas where this might be beneficial are for applying security groups without referencing a security group by ID.

3.Is the concern applying the same "extension" across multiple manifest files? If so I think a pattern like this can be possible:

Yes but also in restricting the scope of the contract between platform and dev teams.