Handling of Multiple Stacks

[skorfmann]

What should be the default for handling multiple stacks within an app?

1. All in the same output directory (well, why have multiple stacks then?)
2. A dedicated folder per stack, totally separated in terms of state
3. A dedicated folder per stack, integrated as modules - monolithic state

[writeameer]

Hi there ,

Does creating multiple stacks work ? Or is this the planning for this feature ?

I have 2 stacks:

const app = new App();
new MyStack(app, 'test1');
app.synth();

const app2 = new App();
new MyStack2(app2, 'test2')
app2.synth();

It only seems to create the second stack.

thanks for any help.

[skorfmann]

Does creating multiple stacks work ? Or is this the planning for this feature ?

That's a feature yet to be implemented. At the moment, the stack file name is static (cdk.tf.json) and therefore overwritten on each synth. This means, the first stack will be overwritten by second one.

@writeameer What's your goal with 2 stacks?

[writeameer]

Hi there @skorfmann ,

Thanks for responding. We use Terraform quite extensively in production. Changes in state, a buggy provider or buggy code can often cause Terraform to drop and recreate a resource. When unintended, this can be catastrophic.

In production, we've broken our deployment up into smaller ones with their own state file to limit the blast radius. In the event an error (failure scenario), we have mitigations in place so the entire data centre doesn't come down all at once.

So being able to have separate state files is quite crucial. In our case, we use Azure.

[skorfmann]

Thanks @writeameer, that looks certainly like the use case I have in mind for multiple stacks as well. This is on our roadmap, but isn't actively worked on at the moment. Right now our focus is on improving some fundamental things around code generation, typing and usability.

• All in the same output directory (well, why have multiple stacks then?)
• dedicated folder per stack, totally separated in terms of state
• dedicated folder per stack, integrated as modules - monolithic state

When I understood you correctly, you'd go for the dedicated folder per stack option here, since this would allow for separated states.

How do you orchestrate your Terraform workflow at the moment, Terragrunt or a custom solution?

[writeameer]

Hi there @skorfmann ,

Thats correct - dedicated folder per stack would work well for us. At this stage, given the maturity of the team, we went with a simpler bespoke solution using bash for driving the orchestration.

[kstewart83]

Would multiple stacks be the right approach in using different profiles with the AwsProvider construct? I need to create different parts of infrastructure with different profiles (as the profiles might be tied to completely different accounts), and it seems like multiple stacks is the first thing that comes to mind. I'm just getting started with the AWS TF CDK and for example I might have something like the below defined:

class MyStack extends TerraformStack {
  constructor(scope: Construct, name: string) {
    super(scope, name);

    new AwsProvider(this, 'aws', {
      region: "us-east-1",
      profile: "profile-a"
    });

    const vpc = new Vpc(this, "awscdk-vpc", {
      tags: {
        Name: "awscdk"
      },
      cidrBlock: "10.0.0.0/16"
    });

    new Subnet(this, "awscdk-subnet", {
      vpcId: Token.asString(vpc.id),
      cidrBlock: "10.0.0.0/24"
    });

  }
}

const app = new App();
new MyStack(app, 'awscdk-terraform');
app.synth();

If I want to create additional infrastructure with "profile-b" for the same project, it's not clear from the examples on how to do that, since referencing the AwsProvider construct in the stack seems to bake it in to the creation of the rest of the constructs. If multiple stacks is the best approach for this (which I think is in-line with the use cases listed above...limited certain profiles in order to limit blast radius), then I'd also be interested in this feature.

[skorfmann]

Would multiple stacks be the right approach in using different profiles with the AwsProvider construct?

While there's no concrete concept for this yet, I'd assume multiple stacks would be the way to tackle this.

Depending on how you handle your state at the moment, and how much you'd be willing to compromise on usability, it's probably possible already.

The main problem is:

• right now the stack is always written to cdktf.out/cdk.tf.json and overwrites whatever was synthesized before

However, you could deploy a dynamically configured stack with changing configuration sequentially. If you'd be using one of the backends you could configure it dynamically.

Here's an example, which would work if you'd want to deploy the exact same stack into different regions:

interface MyStackOptions {
  profile: string;
  region: string;
}

class MyStack extends TerraformStack {
  constructor(scope: Construct, name: string, props: MyStackOptions) {
    super(scope, name);

    const { profile, region } = props;
    
    new S3Backend(this, {
      bucket: 'my-state-bucket',
      key: `cdk/${name}/${region}`
    })

    new AwsProvider(this, 'aws', {
      region,
      profile
    });

    const vpc = new Vpc(this, "awscdk-vpc", {
      tags: {
        Name: "awscdk"
      },
      cidrBlock: "10.0.0.0/16"
    });

    new Subnet(this, "awscdk-subnet", {
      vpcId: Token.asString(vpc.id),
      cidrBlock: "10.0.0.0/24"
    });

  }
}

const app = new App();
new MyStack(app, 'awscdk-terraform', {
  // fetch via ENV or however else you wanna fetch these informations
  profile: process.env.CURRENT_CDK_AWS_PROFILE, 
  region: process.env.CURRENT_CDK_AWS_REGION
});

app.synth();

and then maybe in package.json scripts:

  "scripts": {
    "deploy:ohio": "CURRENT_CDK_AWS_PROFILE=foo CURRENT_CDK_AWS_REGION=us-east-2 cdktf deploy",
    "deploy:frankfurt": "CURRENT_CDK_AWS_PROFILE=bar CURRENT_CDK_AWS_REGION=eu-central-1 cdktf deploy",
    "deploy": "yarn run deploy:ohio && yarn run deploy:frankfurt"
  }

While I haven't actually tried this yet, I don't see a reason why this shouldn't work. However, I don't think that's something I'd seriously pursue for a production stack. It's not parallelizable and it would likely grow out of hand quickly.

Did this answer your question? @kstewart83

[jsteinich]

I haven't tried it yet, but you should also be able to do something like this:

const awsA = new AwsProvider(this, 'awsA', {
  alias: 'a',
  profile: 'a'
});

const awsB = new AwsProvider(this, 'awsB', {
  alias: 'b',
  profile: 'b'
});

new S3Bucket(this, 'bucketA', {
  provider: awsA
});

new S3Bucket(this, 'bucketB', {
  provider: awsB
});

[skorfmann]

@jsteinich sure this will work as well, but is probably more applicable to cases where the same stack spans multiple regions (e.g. this)

[ahoward-conga]

I haven't tried it yet, but you should also be able to do something like this:

const awsA = new AwsProvider(this, 'awsA', {
  alias: 'a',
  profile: 'a'
});

const awsB = new AwsProvider(this, 'awsB', {
  alias: 'b',
  profile: 'b'
});

new S3Bucket(this, 'bucketA', {
  provider: awsA
});

new S3Bucket(this, 'bucketB', {
  provider: awsB
});

@jsteinich sure this will work as well, but is probably more applicable to cases where the same stack spans multiple regions (e.g. this)

This does not seem to work for modules. There is no provider(s) attribute attached to the generated configuration objects for modules such as VPC and EKS.

This makes it not possible to create multiple EKS clusters in different regions with the same stack, unless I am unaware of another way to do this.

@skorfmann Any thoughts other than building out all of the resources directly?

[skorfmann]

This does not seem to work for modules. There is no provider(s) attribute attached to the generated configuration objects for modules such as VPC and EKS.

That's a good point - Could you create a feature request for this?

For the time being, you could resort to an escape hatch on the module object.

[skorfmann]

@jsteinich Since you mentioned that you're using work spaces here #277 (comment)

Do you have an opinion if at all and how workspaces would be modelled in a multiple stacks scenario? I'm putting together an RFC at the moment, and I'm looking for input around work spaces.

[jsteinich]

I see multiple stacks aligning nicely with the logical sections of an application. These sections are likely updated at different cadences, have different risk thresholds (dropping a db is a much bigger concern then restarting an ecs task), have dependencies on other sections, and other factors.

Another dimension (not always independent) is the different environments that an application exists in (dev, qa, staging, prod, etc). Terraform workspaces does pretty well for handling that when it is truly independent. For pieces that are shared between environments, I've split into different applications (terraform state), but that isn't the most convenient.

Mapping things back to terraform, each stack / environment should be a separate state file to ensure blast radius is contained.

We do also use workspaces to handle different instances of the "same" application. So productA and productB use the same infrastructure setup, just different instances of it.

[SizZiKe]

why was this removed from the CDK for Terraform Roadmap? Documentation is still severely lacking and it isn't clear if multiple stacks are supported.

[skorfmann]

why was this removed from the CDK for Terraform Roadmap? Documentation is still severely lacking and it isn't clear if multiple stacks are supported.

Sorry, for the confusion this is certainly misleading. I cleared the roadmap since it was stale and we're doing some housekeeping in Github issues. Multiple stacks is still something which I'd consider as important and I'm pretty sure it'll be back on the roadmap once we've finished the fresh roadmap.

@wolfgz while we're at it: What's your use-cases for multiple stacks and how are you solving it today?

[SizZiKe]

why was this removed from the CDK for Terraform Roadmap? Documentation is still severely lacking and it isn't clear if multiple stacks are supported.

Sorry, for the confusion this is certainly misleading. I cleared the roadmap since it was stale and we're doing some housekeeping in Github issues. Multiple stacks is still something which I'd consider as important and I'm pretty sure it'll be back on the roadmap once we've finished the fresh roadmap.

@wolfgz while we're at it: What's your use-cases for multiple stacks and how are you solving it today?

Thanks for the clarification!

I am not using the CDK yet as I'm looking for the CDK to reach a higher level of adoption and maturity before I begin to refactor our existing modules.

One of the main pain points that I'd like to solve with the CDK is being able to deploy to multiple regions or multiple accounts through dynamic provider references. I realize that this may come to the core Terraform product soon (which I am wholly excited for).

[skorfmann]

FWIW AWS CDK supports multiple stacks in this way: https://docs.aws.amazon.com/cdk/latest/guide/stack_how_to_create_multiple_stacks.html

Yes, I think we'll do something similar as the first iteration of this.

But is there a different recommended way that better utilizes the language constructs provided by cdktf, i.e. "real" type enforcement by TypeScript/Python, as opposed to the light type checking performed in HCL. I was thinking that multiple stacks (i.e. one per environment) might be an approach to this, but since it is not yet supported I'm wondering if people are taking a different approach.

Yeah, multiple stacks would probably be the native cdktf way of doing this. Until then, you could still parameterize your stacks via same variables, driven by env or config files. If that's better than using tfvars depends on the use-case for your variables. Perhaps @jsteinich / @cmclaughlin might have some experience there.

[jsteinich]

Using tfvars works fine if your only changes are things like which instance type or how many of X to create.
Once you get into only create this in prod environment type changes, it becomes rather difficult since support for resource counts is very limited in cdktf at this time. I would definitely recommend using an environment variable to switch which configuration file is used for synth.

I'm not sure if it will be included in with multiple stacks or not, but a simple more built-in solution for different environments will be very valuable.

[skorfmann]

I'm not sure if it will be included in with multiple stacks or not, but a simple more built-in solution for different environments will be very valuable.

Would it help if we would have a first iteration of multiple stacks, where one would have to supply the target stack as argument? That would be similar to what the AWS CDK is doing.

Given the following app:

const app = new App();
new MyStack(app, 'StackA');
new MyStack(app, 'StackB');
app.synth();

The stack would have to be specified for all cli interactions, if there's more than one stack in the app:

cdktf deploy StackA

This would leave the entire orchestration part out for now, but would allow to manage multiple stacks within one app rather easily. Thoughts?

[jsteinich]

Would it help if we would have a first iteration of multiple stacks, where one would have to supply the target stack as argument?

That sounds like a great way to get started. Possibly something that should always be supported as well.

I took another look at how the AWS CDK handles environments and it is more AWS specific than I remembered it being. That said I see a lot of uses where you end up with a partial matrix of stacks made up of the logical sections of the application stack + the different environments it runs in.

[rlmartin]

Once you get into only create this in prod environment type changes, it becomes rather difficult since support for resource counts is very limited in cdktf at this time.

Totally agree. What I'm thinking is more like: the stack is the same, but with env-specific options passed in:

interface StackOptions {
  readonly foo: string
}

const prodConfig: StackOptions = {
  foo: 'bar'
}
const devConfig: StackOptions = {
  foo: 'baz'
}

const app = new App();
new MyStack(app, 'prod', prodConfig);
new MyStack(app, 'dev', devConfig);
app.synth();

then

cdktf deploy prod
cdktf deploy dev

[skorfmann]

Would it help if we would have a first iteration of multiple stacks, where one would have to supply the target stack as argument?

That sounds like a great way to get started. Possibly something that should always be supported as well.

Cool, I think that would be rather straightforward to implement. I agree that this should be supported in general.

[skorfmann]

A few notes for the implementation

• if only one stack present, a cdktf deploy is good enough
• for more than one stack, a cdktf deploy should print found stacks and exit
• for more than one stack, a cdktf deploy prod will deploy the prod stack
• behaviour applies to synth and destroy as well
• each stack will have its own working directory, e.g. cdktf.out/stacks/prod
• there won't be an automatic cleanup of orphaned stacks
• we'll need a way to pass the working directory as context to the stack
• other behaviour should stay as it is (e.g. always overwrite the synthesized file)

[jsteinich]

Should probably match the cli behavior of the AWS CDK. See specifying stacks section of https://docs.aws.amazon.com/cdk/latest/guide/cli.html

[skorfmann]

Should probably match the cli behavior of the AWS CDK. See specifying stacks section of https://docs.aws.amazon.com/cdk/latest/guide/cli.html

Thanks for the link 👍 Being able to provide multiple stacks and patterns is certainly a pretty interesting addition. I'm a bit concerned that something like this cdktf deploy StackA StackB could lead to the assumption, that dependencies between the stacks are somehow resolved automatically. That's something which should happen midterm, but won't be part of the initial implementation. What do you think?

[jsteinich]

I'm a bit concerned that something like this cdktf deploy StackA StackB could lead to the assumption, that dependencies between the stacks are somehow resolved automatically. That's something which should happen midterm, but won't be part of the initial implementation. What do you think?

How will stack dependencies be handled for the first implementation? If they just result in a synth error than deploying multiple stacks at a time shouldn't matter; else, it may not add much scope to order them.

[skorfmann]

I'm a bit concerned that something like this cdktf deploy StackA StackB could lead to the assumption, that dependencies between the stacks are somehow resolved automatically. That's something which should happen midterm, but won't be part of the initial implementation. What do you think?

How will stack dependencies be handled for the first implementation? If they just result in a synth error than deploying multiple stacks at a time shouldn't matter; else, it may not add much scope to order them.

Let's define what dependencies mean in this context. I'm mostly thinking about manually defined dependencies via outputs and remote data sources, since we don't have a way to automatically create cross stack references. Are there other use-cases you have in mind?

[jsteinich]

Are there other use-cases you have in mind?

What happens if a user passes the stack object reference into another stack? Longer term I expect this would create automatic cross stack references, but something needs to happen for the first version; even if just an error.

[skorfmann]

First iteration was released with 0.3

What happens if a user passes the stack object reference into another stack? Longer term I expect this would create automatic cross stack references, but something needs to happen for the first version; even if just an error.

This will be addressed with #651

Deploying all stacks (or a subset) at the same time will be addressed in #650

[Shal10]

Is it possible to have multiple stacks each with different cloud providers?

Code perspective, either having one stack class with conditions based on the "provider" property given while creating a stack class instance before application synthesizing or creating multiple stack classes for each provider?

Code snippet for reference:

import { Construct } from 'constructs';
import { App, TerraformStack } from 'cdktf';
interface CloudProviderConfig {
        provider: string
}
class MyStack extends TerraformStack {
        constructor(scope: Construct, name: string, config: CloudProviderConfig) {
                super(scope, name)
                console.log(config.provider);
                if (config.provider == "aws") {
                        console.log("Provisioning AWS resources: " + config.provider);
                        <<AWS resources provisioning goes here>>
                }
                if (config.provider == "azure") {
                        console.log("Provisioning Azure resources: " + config.provider);
                        <<Azure resources provisioning goes here>>
                }
                if (config.provider == "gcp") {
                        console.log("Provisioning GCP resources: " + config.provider);
                        <<GCP resources provisioning goes here>>
                }
        }
}
const app = new App();
new MyStack(app, 'aws-provider-stack', {
        provider: "aws"
});
new MyStack(app, 'azure-provider-stack', {
        provider: "azure"
});
new MyStack(app, 'gcp-provider-stack', {
        provider: "gcp"
});
app.synth();

When I try to synthesize this code, "cdktf synth" it does not give any error however command runs endlessly.
We are looking for such a feature so, most importantly is this supported yet, or is there a plan for the same in the near future?

It would be great if anyone can throw some light here. :)
@skorfmann @anubhavmishra

[ansgarm]

Hi @Shal10

Is it possible to have multiple stacks each with different cloud providers?

That is already possible, you can mix and match them however you like. There must be some other issue to the code you have that is causing the endless synth. If you can reproduce that in a sharable example it would be helpful if you could file a new bug and share it with us.

[Shal10]

Can you provide any reference links for this scenario - "That is already possible, you can mix and match them however you like"?

I agree, there might be some other issue but as soon as I add two providers to the stack class, synth goes endless.
The code snippet below and is the exact main.ts file I am using and I think it is very basic.

import { Construct } from 'constructs';
import { App, TerraformStack } from 'cdktf';

import * as aws from '@cdktf/provider-aws';
import { AzurermProvider } from './.gen/providers/azurerm';

interface CloudProviderConfig {
        provider: string
}
class MyStack extends TerraformStack {
        constructor(scope: Construct, name: string, config: CloudProviderConfig) {
                super(scope, name);

                console.log(config.provider);

                if (config.provider == "aws") {
                        console.log("Provisioning AWS resources: " + config.provider);
                        new aws.AwsProvider(this, "provider", {
                                region: "us-west-1"
                        });
                }

                if (config.provider == "azure") {
                        console.log("Provisioning Azure resources: " + config.provider);
                        new AzurermProvider(this, "azureFeature", {
                                features: [{}],
                        });
                }

                if (config.provider == "gcp") {
                        console.log("Provisioning GCP resources: " + config.provider);
                }
        }
}

const app = new App();
new MyStack(app, 'aws-provider-stack', {
        provider: "aws"
});
new MyStack(app, 'azure-provider-stack', {
        provider: "azure"
});
new MyStack(app, 'gcp-provider-stack', {
        provider: "gcp"
});
app.synth();

If I make changes to the above code to have only one cloud provider initialized then synth works fine and creates the configuration template for a stack under cdktf.out successfully.

Hope this additional information helps to understand the issue I am trying to explain.

[ansgarm]

Hey @Shal10 👋

I just ran the code you posted and synth finished successfully producing the following files:

> tree cdktf.out                                                                                                                20:31:46
cdktf.out
├── manifest.json
└── stacks
    ├── aws-provider-stack
    │   └── cdk.tf.json
    ├── azure-provider-stack
    │   └── cdk.tf.json
    └── gcp-provider-stack
        └── cdk.tf.json

4 directories, 4 files

Could you create a Github repo with an reproducible example? I suspect something else in your codebase must cause this infite loop. And please file a new issue on this repo linking that repo as it is hard for us to see activity on closed issues.

[skorfmann]

I had a similar issue, where an error happened during synth and it was spinning endlessly. Running the app section of cdktf.json by hand surfaced the error.

[javier-caylent]

Why this issue was closed, I'm trying to find a way to get multipleaccounts in the same project.

[ansgarm]

Hi @javier-caylent!

Why this issue was closed, I'm trying to find a way to get multipleaccounts in the same project.

Because we support multiple stacks since version 0.3 and this issue originally was supposed to track this. The issue was already closed when more comments were added which may have caused this confusion.

Feel free to file a new issue @javier-caylent describing the problem (or just the need for docs) you have.

[javiortizmol]

Hi @javier-caylent!

Why this issue was closed, I'm trying to find a way to get multipleaccounts in the same project.

Because we support multiple stacks since version 0.3 and this issue originally was supposed to track this. The issue was already closed when more comments were added which may have caused this confusion.

Feel free to file a new issue @javier-caylent describing the problem (or just the need for docs) you have.

Gotcha, yes I just saw it, tks :)

[saikirangit]

I am trying to use this multiple stack concept and when I do cdktf synth stack1 it does synth for both stack 1 and stack 2 when It should be just doing stack1? Am I missing anything?

Here is my code

new TerraformPreRequisiteStack( app, 'pre-requisite');

new TerraformStack(app, 'cloud-terraform');

and I am running

cdktf synth pre-requisite

[ansgarm]

Hi @saikirangit!

Synth will always run for all stacks. Actually we can't run synth for only a single stack because only the result from synthesizing will tell us which stacks exist.

When deploying or planning (diff) the cdktf cli will mind the stack that was supplied.

P.S. Feel free to open a new issue next time (also for such questions) - That way we can make sure that we handle every request.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days. This helps our maintainers find and focus on the active issues. If you've found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.