Assemyaml

Assemble and merge multiple YAML sources into a single document.

The goal is to weakly couple YAML documents together for final assembly. For example, a CloudFormation template for a three-tier architecture might contain definitions for individual containers:

Resources:
  ECSTaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      ContainerDefinitions:
        - # Frontend container
          Image: !Ref ReactImage
          PortMappings:
            - ContainerPort: 8080
              HostPort: 80
              Protocol: tcp
        - # Backend container
          Image: !Ref FlaskImage
          PortMappings:
            - ContainerPort: 8080
              HostPort: 1080
              Protocol: tcp
        - # MongoDB container
          Image: !Ref MongoDBImage
          MountPoints:
            - ContainerPath: /opt/mongodb
              SourceVolume: mongodb

This works ok if you're keeping the infrastructure, frontend, backend, and database bits in the same repository. Or you could break everything into separate CloudFormation stacks and hope you get all of the cross-dependencies right.

Neither approach felt right for a simple website being developed by a small but plural number of developers. I wanted to keep the corresponding frontend, backend, and database source for code and infrastructure together and assemble them into a master template (with a few other support pieces in an infrastructure repository).

The above would become four separate documents maintained in separate repositories:

Infrastructure::cfn.yml	React::cfn.yml
Resources: ECSTaskDefinition: Type: AWS::ECS::TaskDefinition Properties: ContainerDefinitions: !Transclude ContainerDefinitions	!Assembly ContainerDefinitions: - # Frontend container Image: !Ref ReactImage PortMappings: - ContainerPort: 8080 HostPort: 80 Protocol: tcp
Flask::cfn.yml	MongoDB::cfn.yml
!Assembly ContainerDefinitions: - # Backend container Image: !Ref FlaskImage PortMappings: - ContainerPort: 8080 HostPort: 1080 Protocol: tcp	!Assembly ContainerDefinitions: - # MongoDB container Image: !Ref MongoDBImage MountPoints: - ContainerPath: /opt/mongodb SourceVolume: mongodb

Syntax

Assemyaml provides two local tags, !Transclude and !Assembly.

The !Transclude tag specifies a transclusion point -- where another document may specify one or more YAML collections (sequence or mapping). It takes a string name used as a label for the transclusion point. The transclusion is a mapping key; if the value is not null, it is used as an assembly for that transclusion point.

The mapping containing the !Transclude tag must not contain any other elements.

The !Assembly tag specifies an assembly -- one or more YAML collections to be injected into a corresponding transclusion point. It takes a string specifying the transclusion label. If multiple documents provide the same assembly, the collection must be the same type; you cannot mix sequences and mappings. If the assemblies are mappings, they must have unique keys.

One document is designated the template. This document is written to the output, with all !Transclude mappings replaced by the assembled values. The other documents are called resources.

Simple examples

Template document	Resource 1	Resource 2	Result
Hello: !Transclude values: - Alpha - Bravo	!Assembly values: - Charlie - Delta	!Assembly values: - Echo - Foxtrot	Hello: - Alpha - Bravo - Charlie - Delta - Echo - Foxtrot

Template document	Resource 1	Resource 2	Result
Hello: !Transclude values: Alpha: 1 Bravo: 2	!Assembly values: Charlie: 3 Delta: 4	!Assembly values: Echo: 5 Foxtrot: 6	Hello: Alpha: 1 Bravo: 2 Charlie: 3 Delta: 4 Echo: 5 Foxtrot: 6

Global Tags

If you're using local tags !Transclude or !Assembly for another purpose (or if local tags offend you), you may tell Assemyaml to use global tags instead:

Template document	Resource document
%TAG !assemyaml! tag:assemyaml.nz,2017: --- Hello: !assemyaml!Transclude values: - Alpha	%TAG !assemyaml! tag:assemyaml.nz,2017: --- !assemyaml!Assembly values: - Bravo

Command-line Usage

assemyaml [options] template-document resource-documents...
assemyaml [options] --template template-document resource-documents...

Options:

• --format json|yaml - Write output in this format. (Only YAML is supported on input.)
• --no-local-tag - Ignore !Transclude and !Assembly local tags and use global tags only.
• --output filename - Write output to filename instead of stdout.

CodePipeline/Lambda Usage

First, create a Lambda function from the Assemyaml ZIP file. Here are three ways of getting the ZIP file:

• Upload a named-version of Assemyaml from S3: https://s3.amazonaws.com/assemyaml.nz/assemyaml-lambda-0.2.1.zip for example.
• Upload the latest version of Assemyaml from S3: https://s3.amazonaws.com/assemyaml.nz/assemyaml-lambda-latest.zip
• Build your own version locally (you'll need Docker):
• • git clone https://github.com/dacut/assemyaml.git
• • cd assemyaml
• • ./lambda-build
• • You'll have a ZIP file named lambda.zip in the current directory.

When used as a Lambda invocation stage in CodePipeline, UserParameters is a JSON object with the following syntax:

{
    "TemplateDocument": "input-artifact::filename",
    "ResourceDocuments": ["input-artifact::filename", ...],
    "DefaultInputFilename": "filename",
    "OutputFilename": "filename",
    "LocalTag": true|false,
    "Format": "yaml|json"
}

All parameters are optional.

TemplateDocument specifies the input artifact and the filename within the artifact to use as the template document.

ResourceDocuments specifies the input artifacts and filename within each artifact to use as resource documents. Any input artifacts not referenced in either TemplateDocuments or ResourceDocuments are appended to ResourceDocuments as artifact::DefaultInputFilename.

The DefaultInputFilename key is used for an input artifact filename if an input artifact is not referenced in either TemplateDocument or ResourceDocuments. It defaults to assemble.yml.

OutputFilename specifies the filename to write in the output artifact. It defaults to assemble.yml.

LocalTag specifies whether the !Transclude and !Assembly local tags are allowed. It defaults to true.

Format specifies the output template format. It defaults to yaml.

If TemplateDocument or ResourceDocument is not specified, the following behavior applies:

Options specified	Input artifacts: `[A, B, C]`
{ "TemplateDocument": "B::f2", "ResourceDocuments": [ "A::f1", "C::f3" ] }	{ "TemplateDocument": "B::f2", "ResourceDocuments": [ "A::f1", "C::f3" ] }
{ "TemplateDocument": "B::f2" }	{ "TemplateDocument": "B::f2", "ResourceDocuments": [ "A::assemble.yml", "C::assemble.yml" ] }
{ "ResourceDocuments": [ "A::f1", "C::f3" ] }	{ "TemplateDocument": "B::assemble.yml", "ResourceDocuments": [ "A::f1", "C::f3" ] }
{ "ResourceDocuments": [ "C::f3" ] }	{ "TemplateDocument": "A::assemble.yml", "ResourceDocuments": [ "C::f3", "B::assemble.yml" ] }
	{ "TemplateDocument": "A::assemble.yml", "ResourceDocuments": [ "B::assemble.yml", "C::assemble.yml" ] }