Handling of AWS CloudFormation intrinsic functions in 0.1.0

[PaulMaddox]

Problem Statement

We need to be able to unmarshal AWS CloudFormation resources to typed Go structs, for example:

type AWSLambdaFunction struct {
    Runtime string 
    ...
}

This is complicated, as within a CloudFormation template, a property (such as Runtime) could be a string, or it could be an AWS CloudFormation intrinsic function, such as:

MyLambdaFunction:
    Type: AWS::Serverless::Function
    Properties:
        Runtime: !Join [ "nodejs", "6.10" ]

The above example would fail to unmarshal into the AWSLambdaFunction struct above with json.Unmarshal() or yaml.Unmarshal().

Considerations

• Customers can nest intrinsic functions.
• We don't want to necessarily resolve intrinsic functions, but we should ensure that any intrinsic function is replaced with a primitive object (string).
• Intrinsic functions can be specified in YAML as a tag (e.g. !Sub), which isn't supported in the Go YAML parser today (they get replaced with blank strings).

Implementation

I propose we:

1. Submit a patch to the Go YAML parser to support tags, and provide a callback where the parser user choose what to replace for each tag.

2. We unmarhal all YAML templates into interface{}, then into JSON for handling within Goformation. This allows us to just focus on supporting a single template language in Goformation code.

I propose we implement a decoupled intrinsics handling package with a very simple interface that takes a JSON AWS CloudFormation template (as a []byte), processes all intrinsics, and returns the resulting JSON template (as a []byte).

This package will have a very simple API. Here is a usage example:

package main

import (
	"fmt"

	"github.com/paulmaddox/intrinsic-proposal/intrinsics"
)

const template = `{
	"Resources": {
		"ExampleResource": {
			"Type": "AWS::Example::Resource",
			"Properties": {
				"SimpleProperty": "Simple string example",
				"IntrinsicProperty": { "Fn::Join": [ "some", "name" ] },				
				"NestedIntrinsicProperty": { "Fn::Join": [ "some", { "Fn::Join": [ "joined", "value" ] } ] }
			}
		}
	}
}`

func main() {

	processed, err := intrinsics.Process([]byte(template))
	if err != nil {
		fmt.Printf("ERROR: Failed to process AWS CloudFormation intrinsics: %s\n", err)
	}

	fmt.Print(string(processed))

	// Go on to perform normal template unmarshalling here
	// ...

}

The resulting output of the above example would be:

{
  "Resources": {
    "ExampleResource": {
      "Type": "AWS::Example::Resource",
      "Properties": {
        "SimpleProperty": "Simple string example",
        "IntrinsicProperty": "-- refused to resolve Fn::Join intrinsic function --",
        "NestedIntrinsicProperty": "-- refused to resolve Fn::Join intrinsic function --"
      }
    }
  }
}

Below is an implementation of the intrinsic handling package I am proposing.
It's pretty simple, and recurses through a JSON structure (interface{}), looking for intrinsic functions. If it finds any, it calls out to a hook function that does whatever it needs to in order to "resolve" the intrinsic function back from an object, into a simple primitive.

This hook approach would allow us the possibility of resolving some intrinsic functions in the future if we wanted to, however in the initial implementation we would simply refuse to resolve any of the intrinsic functions, and just return a string value such as "unsupported intrinsic function: Fn::Join".

package intrinsics

import (
	"encoding/json"
	"fmt"
)

// IntrinsicHandler is a function that applies an intrinsic function  and returns the
// response that should be placed in the object in it's place. An intrinsic handler
// function is passed the name of the intrinsic function (e.g. Fn::Join), and the object
// to apply it to (as an interface{}), and should return the resolved object (as an interface{}).
type intrinsicHandler func(string, interface{}) interface{}

// IntrinsicFunctionHandlers is a map of all the possible AWS CloudFormation intrinsic
// functions, and a handler function that is invoked to resolve.
var intrinsicFunctionHandlers = map[string]intrinsicHandler{
	"Fn::Base64":      nonResolvingHandler,
	"Fn::And":         nonResolvingHandler,
	"Fn::Equals":      nonResolvingHandler,
	"Fn::If":          nonResolvingHandler,
	"Fn::Not":         nonResolvingHandler,
	"Fn::Or":          nonResolvingHandler,
	"Fn::FindInMap":   nonResolvingHandler,
	"Fn::GetAtt":      nonResolvingHandler,
	"Fn::GetAZs":      nonResolvingHandler,
	"Fn::ImportValue": nonResolvingHandler,
	"Fn::Join":        nonResolvingHandler,
	"Fn::Select":      nonResolvingHandler,
	"Fn::Split":       nonResolvingHandler,
	"Fn::Sub":         nonResolvingHandler,
	"Ref":             nonResolvingHandler,
}

// nonResolvingHandler is a simple example of an intrinsic function handler function
// that refuses to resolve any intrinsic functions, and just returns a basic string.
func nonResolvingHandler(name string, input interface{}) interface{} {
	result := fmt.Sprintf("-- refused to resolve %s intrinsic function --", name)
	return result
}

// Process recursively searches through a byte array for all AWS CloudFormation
//  intrinsic functions,
// resolves them, and then returns the resulting interface{} object.
func Process(input []byte) ([]byte, error) {

	// First, unmarshal the JSON to a generic interface{} type
	var unmarshalled interface{}
	if err := json.Unmarshal(input, &unmarshalled); err != nil {
		return nil, fmt.Errorf("invalid JSON: %s", err)
	}

	// Process all of the intrinsic functions
	processed := search(unmarshalled)

	// And return the result back as a []byte of JSON
	result, err := json.MarshalIndent(processed, "", "\t")
	if err != nil {
		return nil, fmt.Errorf("invalid JSON: %s", err)
	}

	return result, nil

}

// Search is a recursive function, that will search through an interface{} looking for
// an intrinsic function. If it finds one, it calls the provided handler function, passing
// it the type of intrinsic function (e.g. 'Fn::Join'), and the contents. The intrinsic
// handler is expected to return the value that is supposed to be there.
func search(input interface{}) interface{} {

	switch value := input.(type) {

	case map[string]interface{}:

		// We've found an object in the JSON, it might be an intrinsic, it might not.
		// To check, we need to see if it contains a specific key that matches the name
		// of an intrinsic function. As golang maps do not guarentee ordering, we need
		// to check every key, not just the first.
		processed := map[string]interface{}{}
		for key, val := range value {

			// See if we have an intrinsic handler function for this object key
			if handler, ok := intrinsicFunctionHandlers[key]; ok {
				// This is an intrinsic function, so replace the intrinsic function object
				// with the result of calling the intrinsic function handler for this type
				return handler(key, search(val))
			}

			// This is not an intrinsic function, recurse through it normally
			processed[key] = search(val)

		}
		return processed

	case []interface{}:

		// We found an array in the JSON - recurse through it's elements looking for intrinsic functions
		processed := []interface{}{}
		for _, val := range value {
			processed = append(processed, search(val))
		}
		return processed

	case nil:
		return value
	case bool:
		return value
	case float64:
		return value
	case string:
		return value
	default:
		return nil

	}

}

Reference Information

Possible intrinsic function formats (JSON):

List of intrinsic functions:

• Fn::Base64
• Fn::And
• Fn::Equals
• Fn::If
• Fn::Not
• Fn::Or
• Fn::FindInMap
• Fn::GetAtt
• Fn::GetAZs
• Fn::ImportValue
• Fn::Join
• Fn::Select
• Fn::Split
• Fn::Sub
• Ref

[pesama]

+1. Simpler than our initial approach to intrinsics, removing all type assertions. I agree with doing it iteratively, starting by not supporting them and resolve to a predefined value. Also agreed to parse YAML to JSON before goformation handles the code. I like this approach Paul.

Would we consider the tag parsing feature in YAML's parser as a requirement for this, or can we iterate and release in parallel?

[sanathkr]

+1 agreed with proposal. Two questions:

1. Why convert to JSON and recourse? If the Yaml parser has callback for intrinsics, why do we need a separate recursive parser?

Goformation's main method must take a intrinsic parser object class as input. Today there is only one parser. If needed customers can extend this with new parsers that does more magic like resolve references to parameters etc.

[pesama]

The YAML parser only would have a callback for inline intrinsics - i.e. YAML tags, like !Sub - but not for the other intrinsic functions. Once the first parsing substitutes the tags, we marshal back to JSON - independently of the original language - to only handle final template unmarshalling in one language.

[sanathkr]

Okay. Why to JSON is my question. Why can't you unmarshall to a generic map for recursion?

Why YAML -> JSON -> map when you can Yaml -> map?

[PaulMaddox]

Good point. The input and output for the intrinsic processor can both just be interface{}.

[PaulMaddox]

Actually, that doesn't help much.

After intrinsic processing, the next step will be to unmarshal the processed output into Go structs.

Rather than do that ourselves, it's easier to use the JSON unmarshaller. Hence why []byte of JSON made more sense as an output.

[pesama]

The full flows would be YAML->map->JSON->(parsed template) for YAML templates, and JSON->(parsed template) for JSON templates, right?

[sanathkr]

Okay so here is the flow for both:

YAML -> handle !Ref style tags -> convert intrinsic objects to primitive types -> marshall to JSON -> unmarshall to Go Structs

JSON -> map -> convert intrinsic objects to primitive types -> marshall to JSON -> unmarshall to Go Structs

Is my understanding correct?

I will take up with work to patch YAML parser to handle !Ref style tags. One of you can begin converting intrinsics to primitive types. Both look independent.

Let's have Unit tests, unit tests, and unit tests!

[PaulMaddox]

Yes, that's the correct flow. And just to be clear, "handle !Ref style tags" means convert them from short form (tag), into long form?

i.e.

!Join [ delimiter, [ comma-delimited list of values ] ]
becomes
Fn::Join: [ delimiter, [ comma-delimited list of values ] ]

I'll work on a PR for the above proposal.
@sanathkr - when you've submitted the PR for the YAML library, can you link to it here.

[sanathkr]

Yes, you're right. Convert short form to long form. At which point it becomes just another object.

I'll link it here once I have a PR out