[Proposal] Support for environment-specific addons

[adrienlauer]

This is a proposal to fix issue #1975 regarding the ability to create environment-specific resources. These resources would be shared across all services.

An _addons subdirectory containing environment add-ons would be created in the copilot directory. This would contain addons that apply to all environments. To specify an addon for a specific environment, a subdirectory named after the environment would be created.

Example:

<project-root>
    |- copilot/
        |- _addons/
            |- addon-1.yml
            |- addon-2.yml
            |- dev/
                |- dev-addon-1.yml
            |- prod/
                |- prod-addon-1.yml
        |- service1/
            |- addons/
                |- service-addon-1.yml
            |- manifest.yml

Environment resources would be created automatically when the first service is deployed. For a specific environment, all relevant add-ons would be merged into a unique CF stack, deployed before the service add-ons stack.

Considering the example above, the copilot svc deploy -e dev -n service1 would execute the following steps:

1. The environment add-ons CF stack, containing addon-1.yml, addon-2.yml and dev-addon-1.yml, is deployed.
2. The service1 add-ons CF stack, containing service-addon-1.yml, is deployed as usual.
3. The service1 CF stack is deployed as usual.

Environment add-ons resources would be deleted only when the environment itself is deleted.

This solution seems to nicely integrate in the current copilot workflow. Some commands could be added in the future (like copilot env update) to improve upon this solution.

I am proposing this with @rverpillot and we are reasonably confident to be able to propose a PR, so feel free to criticize and improve upon this idea. Obviously, we'll start working on this only if approved by the team. Thanks !

[efekarakus]

Bonjour @adrienlauer and @rverpillot! 😄

Thank you so much for the awesome proposal 🙇. Before we proceed would you mind expanding on what environment specific resources you'd like to create in your applications?
I want to make sure we provide the right integration points so that it's easy to connect a service with an environment addon.

For example: "I'd like to create an EventBus as an environment addon so that 'Worker Service's can consume events with EventRules".

In the mean time, I can share also another draft proposal for environment addons: https://gist.github.com/efekarakus/e4d6f798138d30da60b069a7d5498026

We'd love to hear y'alls feedback on the design section above!

[rverpillot]

Bonjour @efekarakus

We have to create these resources:

• private endpoints connected to on premise services (connections could be different regarding the environment).
• private hosted zones.
• DNS records.
• WAF rules.
• Maybe specific ALB, not internet facing and/or created in another ingress VPC.
• ...

We're evaluating the migration of many projects to ECS, and this list of resources could grow.

The lifecycle of these resources is the same of environment, so using environment addons seems the right solution. Today we are using a mix of CDK and AWS CLI commands, or service addons (but I don't like to bound shared resources to a dedicated service).

Nice to see that a draft design already exists. We are looking forward to reading it and doing feedback.

Is documentation available about the internal design of copilot-cli ?

[efekarakus]

Apologies for the late response 🙇

(but I don't like to bound shared resources to a dedicated service).

Yup totally fair!

Is documentation available about the internal design of copilot-cli ?

Ah unfortunately we don't have anything like that 😭 In general, our approach has been to submit proposal designs when ready as a github issue to collect feedback: #2918

So there are a lot of small issues like this scattered around the repository, but not one big design doc on how the internals work. That'd be a cool talk to do I think.

[adrienlauer]

Hi @efekarakus and thank you for your feedback. Sorry too for the late reply.

I and @rverpillot discussed about your draft proposal and found it very interesting. It seems to open the door to a more generic structure, using manifest for various concepts instead of only services.

In that spirit maybe we can go a bit further and name the new manifest type Environment instead of Environment Addons. This would allow to specify environment future settings directly in the manifest in addition to the ability to put related add-ons in the addons subdirectory.

Regardless of the manifest type, one of our need is to specify different add-ons for each environment. The basic solution to this in our initial idea was to create subdirectories inside addons named after the environment they apply to. This could also apply to your proposal (and to services too), but I feel that's a separate issue that we'll discuss in another ticket.

One worry we have with your proposal, is that it seems more difficult to implement than what we planned for ours. Do you have an initial implementation of it yet ? Or maybe you can share some ideas about it so we can have a clearer view of what needs to be done.

Thanks!

[tmichalski]

I don't have anything to add to the architecture proposed as I see that the needs of my organization will be met with Environment Addons. We wouldn't need the environment-specific addons, but I can see how they could be useful like environment-specific variables.

A use case that I would like to share within my organization is that we have a shared AWS Cognito User Pool per environment. Currently we create the Cognito User Pool in one of the services as an addon, then export the UserPoolId generated, and then import the UserPoolId into the backend microservices that need it (some micro services re-validate JWT tokens in headers). We have to document in a root-project level DEPLOY_README file that the services need to be deployed in a specific order with the service containing the Cognito addon deployed first.

It's not ideal to have to document the deploy order, but it's also not intuitive to have the AWS Cognito resource in a service that doesn't technically "own" that resource. So we have to communicate to developers who need to change the CF addon config where to find the Cognito addon.

Having the AWS Cognito resource created at the environment level would allow us to make the UserPoolId variable available to any of our resources without having to define a deploy order.

[matthewhembree]

I'm possibly misunderstanding the title.

"Environment-level" resources as defined by #1975 are those resources that would be provisioned at environment init.

This is referring to "environment-level" add-on resources with configuration "specific" to a particular environment. Provisioned at environment init AND specific to an environment.

Is this necessary if #1975 is present? Would a Cloudformation conditional placed on the resource based on the Env parameter be sufficient?

e.g.

# Master environment-level add-ons file.
Conditions:
  IsDev: !Equals [ !Ref Env, dev ]
  IsProd: !Equals [ !Ref Env, prod ]

Resources:
  MyResource:
    Type: AWS::SomeService::SomeFeature
    Condition: IsDev

[efekarakus]

Hi folks! @Lou1415926 posted a proposal to support env addons 🚀 #4219 we would love to hear your feedback!

[Lou1415926]

Hello all! The environment addons feature is now released in v1.25.0 🎉🚀!