Support new YAML template features

[robert2d]

CloudFormation Now Supports YAML.

Looks like stackup will need some design and then development given its strict JSON template handling: https://github.com/realestate-com-au/stackup/blob/master/lib/stackup/stack.rb#L115 (as an example)

Im going to have a look into converting a small stack in the next week or so, and will try attempt a YAML stackup refactor with that stack in both JSON and YAML. I'll at least try and provide any relevant info if and when possible.

References

Jeff's blog and release notes:
https://aws.amazon.com/blogs/aws/aws-cloudformation-update-yaml-cross-stack-references-simplified-substitution/

CloudFormation Formats documentation:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-formats.html

[mdub]

Actually, stackup has always supported templates (and parameters) in YAML format. Internally, it's just data, which we then serialise to JSON when invoking the CloudFormation API.

I see no advantage in uploading the original YAML to CloudFormation, do you?

[robert2d]

Perfect, thanks Mike.

[mdub]

I've reopened this because, while we already support YAML, we don't support the new "abbreviated syntax for tags", e.g. !GetAtt vs Fn::GetAtt.

[lukeck]

I'm not sure how much of an advantage it is but if a template is uploaded in YAML, get-template gives it back to you in YAML.

[mdub]

@lukeck: Damn. If that's true it means we probably can't away with converting the template into a canonical form (i.e. data) within Stackup; instead, we'll have to retain the original template text.

[tigris]

This thread is relevant to my interests.

[mdub]

I'm kind of pissed at the way AWS have chosen to use (abuse?) YAML tags, in their adoption of YAML, e.g.

Outputs:
  LoadBalancerAddress:
    Value: !GetAtt
    - LoadBalancer
    - DNSName

as a shorthand for

Outputs:
  LoadBalancerAddress:
    Value: 
      Fn::GetAtt:
      - LoadBalancer
      - DNSName

As I understand it, tags are intended to denote types. The really annoying thing is that the first example above doesn't even round-trip via YAML!

irb> puts YAML.dump(YAML.load(raw))
---
Outputs:
  LoadBalancerAddress:
    Value:
    - LoadBalancer
    - DNSName

FFS! I wish they'd just treated YAML as an alternative serialisation format.

Anyway, If we want Stackup to support the new syntax, we have two options:

A. retain input template format/body (awscli-compatible)
B. parse-time conversion to data (backward-compatible)

Option A: retain input template format

To be fully compatible with what the AWS CLI (now) does, we'd have to retain the original YAML template text, and send it intact up to CloudFormation. As a result:

• AWS would take take of the funky tag handling
• we wouldn't lose tags used in the input YAML
• Stackup's behaviour would change (where we now send JSON, we'd be sending YAML)
• Stackup internals would have to change (because we currently treat templates as Ruby data)
• we would no longer be "normalising" the template at load time, so "diffs" would get funky, if either the format (JSON/YAML) for formatting (e.g. amount of whitespace) changed

Option B: parse-time conversion

A simpler change is to stick with our current approach - converting the input JSON/YAML to data within Stackup - and just add some magic to convert the tags to their equivalent JSON-compatible forms, e.g. !GetAtt to Fn::GetAtt. And then:

• most of Stackup would be unchanged
• diffs would continue to work
• bit it's a bit lossy, as tags are converted to their more verbose form
• going forward, we would have to match any other "YAML magic" AWS chose to implement

[lukeck]

I'm not in love with option A but option B looks like it's a recipe for an ongoing headache.

[mdub]

I'm not sure it's that much of an ongoing headache, @lukeck. The logic I've implemented in #32 is basically:

• if you see !Ref, substitute Ref
• if you see !SomethingElse, substitute Fn::SomethingElse

See

stackup/lib/stackup/yaml.rb

Lines 46 to 55 in 3b028a6

	def accept(target)
	case target.tag
	when "!Ref"
	{ "Ref" => super }
	when /^!(\w+)$/
	{ "Fn::#{$1}" => super }
	else
	super
	end
	end

[mdub]

Closed with #32.

[benhutchison]

Stackup's implementation doesn't seem to support YAML AWS pseudo-parameter substitution via !Sub .

  SubscribeLambda:
    Type: "AWS::SNS::Subscription"
    Properties:
      Endpoint: !Sub |
        arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:addBuyProfile
      Protocol: lambda
      TopicArn: !Sub |
        arn:aws:sns:${AWS::Region}:${AWS::AccountId}:cbr-profiler

I concur with @lukeck assessment that option (b) will be a headache, due to this sort of thing

[benhutchison]

Additionally, there seems to be a trailing newlines that gets inserted into ARN strings in YAML, making them fail AWS validation

Template:

Resources:
  SnsPermission:
    Type: "AWS::Lambda::Permission"
    Properties:
      Action: lambda:InvokeFunction
      FunctionName: !Sub |
        arn:aws:lambda:ap-southeast-2:123456789012:function:myLambda
      Principal: sns.amazonaws.com
      SourceArn: !Sub |
        arn:aws:sns:ap-southeast-2:123456789012:*

Result (note the newline that has been retained at end of ARN is whats failing the regex validation)

INFO: [15:35:50] SnsPermission - CREATE_FAILED - 2 validation errors detected: Value 'arn:aws:sns:ap-southeast-2:123456789012:*
' at 'sourceArn' failed to satisfy constraint: Member must satisfy regular expression pattern: arn:aws:([a-zA-Z0-9\-])+:([a-z]{2}-[a-z]+-\d{1})?:(\d{12})?:(.*)

[mdub]

It should support !Sub, @benhutchison. I think the problem is that by using |, you've got newlines in the source YAML. Try without them, e.g.

 SubscribeLambda:
    Type: "AWS::SNS::Subscription"
    Properties:
      Endpoint: !Sub arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:addBuyProfile
      Protocol: lambda
      TopicArn: !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:cbr-profiler

If that still doesn't work, please raise a separate issue.