cloudformation-include: CfnInclude multiple files into one template in CDK v2

[mxl519]

Describe the feature

The CfnInclude construct from CDK v1's @aws-cdk/core allowed the import of multiple CloudFormation templates into a stack, with the official CDK docs)describing the result as "All elements of the template will be merged into the current stack, together with any elements created programmatically." Effectively this resulted in the final CloudFormation template being the input templates concatenated in the order that they were listed in the CfnInclude.

This merge functionality seems to have disappeared once CfnInclude got deprecated from @aws-cdk/core in favour of @aws-cdk/cloudformation, which is now the only CfnInclude construct available in CDK v2. The CDK docs don't explicitly mention it, but CfnInclude with cross-template dependencies no longer work because each template is validated independently, and we see CDK errors when attempting to build those resources.

Use Case

For our use case, we import a CFN nested stack using a pair of YAML template files owned by a partner team for connecting to their VPCE. They have their template files modularized, so one file contains just a Mappings section, that is then referenced by other template files. The simplified version of it is something like:

mappings.template.yml

---
Mappings:
  EndpointsMap:
    us-east-1: com.amazonaws.vpce.us-east-1.vpce-svc-01234567890123456
    us-west-1: com.amazonaws.vpce.us-west-1.vpce-svc-01234567890123457

endpoint.template.yml

---
AWSTemplateFormatVersion: "2010-09-09"

Parameters:
  VPC:
    Type: String

Resources:
  ServiceVPCEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      VpcEndpointType: Interface
      ServiceName: !FindInMap
        - EndpointsMap
        - !Ref 'AWS::Region'
      VpcId: !Ref VPC
...(rest of it isn't relevant)

We were able to import them into our nested stack with CDK v1 like so:

import { CfnInclude } from 'monocdk';

const nestedResources: NestedStack = new NestedStack(this, 'NestedResources');

new CfnInclude(nestedStack, 'mappings', { template: fs.readFileSync('file/path/to/mappings.template.yml', 'utf8') });
new CfnInclude(nestedStack, 'endpoint', { template: fs.readFileSync('file/path/to/endpoint.template.yml', 'utf8') });

However, this is no longer possible after migrating to CDK v2, as we get the following build error:
Error: Mapping used in FindInMap expression with name 'EndpointsMap' was not found in the template. Additionally, since the type of CfnIncludeProps in @aws-cdk/cloudformation has changed to take in templateFile as the name of the template file, instead of template as the contents of the file, we're not able to work around this very easily by reading the files in together and concatenating the strings.

Proposed Solution

Ideally, it'd be great to have some optionality to merge all files passed in via CfnInclude into the stack, as it worked in @aws-cdk_core.CfnInclude.

Beyond that, allowing us to pass in the contents of the template file via template in CfnIncludeProps could also work. Someone had also suggested a stopgap solution of reading in the files, writing out a temporary template YAML file by concatenation, and then passing in that temporary file name into a single CfnInclude parameter, but we were hoping for a more straightforward way to do it.

Finally, I think maybe if we nested the Mappings resource into a nested nested stack, we could reference the mappings as an Output from the nested EndpointService stack, but we're not sure if that's even a good solution, as nesting stacks multiple levels deep just for a mapping seems like overkill.

Other Information

Apologies for any typos in the samples, I tried to take our actual templates as an example but there was a lot of data to redact, hopefully the concept of our issue is clear. Thanks!

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CDK version used

1.177.0, 2.73.0

Environment details (OS name and version, etc.)

MacOS Ventura 13.3.1, Amazon Linux 2 x86_64

[pahud]

Thank you for your report.

Are you able to use the templateFile property for CfnInclude in CDK v2? What error would you see with that?

[mxl519]

@pahud Yes, we pass in the templateFile property when trying to upgrade to CDK v2:

import { CfnInclude } from 'aws-cdk-lib/cloudformation-include';

const nestedResources: NestedStack = new NestedStack(this, 'NestedResources');

new CfnInclude(nestedStack, 'mappings', { templateFile: 'file/path/to/mappings.template.yml' });
new CfnInclude(nestedStack, 'endpoint', { templateFile: 'file/path/to/endpoint.template.yml' });

As far as I can tell, we don't have any other option anyways, as that is a required parameter. The error message we see is Error: Mapping used in FindInMap expression with name 'EndpointsMap' was not found in the template, and the stack trace points to the second CfnInclude line.

[scanlonp]

@mxl519 Is that the exact format of the mappings template? If so, there may be a layer missing to the mapping that could cause the error message you are seeing.
i.e.

Mappings:
  EndpointsMap:
    us-east-1: 
      endpoint: com.amazonaws.vpce.us-east-1.vpce-svc-01234567890123456
    us-west-1: 
      endpoint: com.amazonaws.vpce.us-west-1.vpce-svc-01234567890123457

If not, could you try to create the mapping in your stack and then include the endpoint.templaye.yml file?

Something like

import { CfnInclude } from 'aws-cdk-lib/cloudformation-include';
import * as cdk from 'aws-cdk-lib/core'

const nestedResources: NestedStack = new NestedStack(this, 'NestedResources');

const endMap = {
  us-east-1: { endpoint: 'com.amazonaws.vpce.us-east-1.vpce-svc-01234567890123456' },
  us-west-1: { endpoint: 'com.amazonaws.vpce.us-west-1.vpce-svc-01234567890123457' },
};

new cdk.CfnMapping(this, 'EndpointsMap', {
  mapping: endMap,
});

new CfnInclude(nestedStack, 'endpoint', { templateFile: 'file/path/to/endpoint.template.yml' });

(The code might not be flawless but you have the idea).

Then if you are still getting the error it may be the include function.

I know this is not the cleanest long term solution since reading in the template is much preferable to making the map object in the code by hand, but it may pinpoint your error.