CloudFormation Resources

[ktheory]

I took a stab at adding resources and a waiter for AWS CloudFormation.

(Modeling all these APIs as data instead of code is awesome, btw. 🍰 👍 )

Here's what this PR adds:

cfn = Aws::CloudFormation::Resource.new
stacks = cfn.stacks.to_a # List your stacks

# Create a new stack
# try launching https://s3-us-west-2.amazonaws.com/cloudformation-templates-us-west-2/SQS_With_CloudWatch_Alarms.template
# as a template that only creates free resources

stack = cfn.create_stack(stack_name: '...', template_url: '...')

# Wait for it to be ready
stack.wait_until_complete

# List the stack events
stack.events

# List the stack resources
stack.stack_resources

# Modify the stack
stack.update(template_url: '...', parameters: [...])

# And wait until that's done
stack.wait_until_complete

# Shut it down
stack.delete

Questions/comments/gotchas

I closely copied other *.resources.json* files as examples to figure out how resources work.

What is a subResource?

I wasn't sure how subResources differed from hasOne or hasMany resources, or just a resource action that returns structs.

How to initialize resources with more attributes than their identifiers?

StackResources are populated by the DescribeStackResources API call, which returns an array of fully hydrated StackResources. But I could only figure out how to pass the identifiers to the StackResources collection. So each StackResource makes a redundant API call to load the rest of it's attributes. Any way to avoid that?

Tests???

I'd like to add some unit tests for the features I added. But I don't see any specs/integration tests for any resources. 😬 Did I miss them?

[ktheory]

NB: I tried naming this just Resource, but that caused warnings about redefining a constant. So I went with the more verbose StackResource.

[trevorrowe]

By convention, the Ruby SDK generates a Resource class for each service module. That is where the name conflict comes from. I agree with the rename.

[coveralls]

Coverage remained the same when pulling 52931db on ktheory:cloudformation_resources into b3ed6fa on aws:master.

[ktheory]

I didn't make Events a full hasMany resource since you can't interact with them. It's just an action that returns structs.

@trevorrowe Maybe I should do the same thing for StackResources?

[ktheory]

FYI, I made Events a proper resource instead of an action in 9d73027.

[trevorrowe]

What is a subResource?

A sub-resource is a child resource to another resource. For example, an object in S3 is the child of a bucket. You can not identify the object without also identifying its parent.

A has one or has some association is a reference from one resource to another. For example, an EC2 instance "has some" security groups. Neither security group or instance is a parent/child of the other, but they are associated.

In the current implementation, a has one/some association creates optional associations based on the presence of the associated resource id. If an EC2 instance does not have a subnet id, it's "has one" subnet reference would return nil.

From a child resource, you can always get a reference to its parent.

Also, child resources do not get helper methods from the service resource, for example:

# S3 Bucket is not a sub resource, so there is a #bucket method of S3::Resource
s3.bucket(bucket_name)

# S3 Object *is* a sub resource, so you can only refernece it from a bucket
s3.object(key) # NoMethodError
s3.bucket('name').object('key') # works

How to initialize resources with more attributes than their identifiers?

When you define the hasMany association from Stack to StackResource, you must specify the "operation" and the "resource" (which you have). These define what API call to make, and how to construct resource objects from the response. There is one more optional key, that you are missing, called "path". If you give a path expression that resolves to a list, this is treated as a path to the resource "shape" or data. These objects will be given to the resource objects constructors as :data. This removes the need for n+1 describe calls.

Unfortunately, not all APIs provide a list operation that returns data. Some return only identifiers, so the "path" is optional in the definition.

Tests???

Currently, there is a work-in-progress resource definition linter. Once complete, this should validate your resource definition. It currently validates things such as:

• operations named exist in the API model
• params name exist as valid targets
• types match
• paths resolve to the proper shape
• etc.

There are many more things that can be linted on the resource definition. I hope to fill out the validator more. Unfortunately, this can tell if you define a "Update" method and copy and paste the name of the delete method. It will only tell you that you are correctly calling the delete api. :)

For this reason, some form of tests are needed, but I haven't decided in what shape or form these tests should happen. Integration tests would be awesome, but they tend to be slow and expensive, which prevents them from running very often. I'm open to suggestions. There are some fairly extensive functional tests that show how a resource definition is translated into Ruby land classes and methods. These are not complete or exhaustive yet, but I hope to continue fleshing them out.

Testing is the primary reason why these interfaces are in a "preview" state. I don't expect to seem many changes in how the user interacts with them, but there may be some changes to the definition / schema. Certainly there needs to be some guide-level documentation that accompanies the JSON schema and expanded linter tests. Additionally, I would like to expand the number of services with coverage to ensure they are flexible enough to cover the 90% case of API patterns.

[trevorrowe]

Also, I wanted to say that I think it's awesome that you figured this out and started adding definitions for an additional service. You may not know this, but these definitions are being shared between Ruby and the new Java SDK abstractions here: http://aws.amazon.com/blogs/aws/java-sdk-resource-apis-preview/

They obviously consume the document differently that we do in Ruby, but they share the same definitions. So efforts here benefit many other developers. Also, there is a work-in-progress implementation in Python, and I wouldn't be surprised to see implementations in other languages.

[trevorrowe]

Lastly, I haven't had a change to review this yet, I'll try to dig into it and give feedback tomorrow.

[ktheory]

@trevorrowe Thanks for explaining so quickly. Neat that this will be used in SDKs for other languages too. 😄.

As for the testing, your linter suggestion sounds great to cover the most common bugs.

[ktheory]

I went with Events for terseness. Could be StackEvents to parallel StackResources. ¯(°_o)/¯

[trevorrowe]

I pushed a commit earlier today that exposes the work-in-progress linter as a rake task. Running rake resources:validate will lint all of the resource definitions in apis/. As I mentioned earlier, the linter is not complete. That said, it kicked up two errors:

• '#/resources/StackResource/load/path' did not resolve to a 'StackResource' shape.
• '#/resources/Event/identifiers/0/name' must not be prefixed with 'Event'.

The 2nd error requires that resources do not prefix their identifiers with their name. This avoids methods such as Event#event_id. Instead you would have Event#id.

The first error is a bit more complex. By design, the resource definitions require that any "path" expression that provides resource data resolves to the shape referenced as "shape". The "shape" defines what attributes the resource has and it ensures you don't have to deal with partial hydration.

The problem with this API is that Cloud Formation has three different descriptions of a stack resource:

• StackResourceSummary - as returned by ListStackResources
• StackResource - as returned by DescribeStackResources
• StackResourceDetail - as returned by DescribeStackResource

StackResource and StackResourceDetail appear to be identical except the detail shape has a Metadata member, and it also renames Timestamp to LastUpdatedTimestamp.

Looking more at the various APIs, there are a few other differences:

• ListStackResources is pageable
• DescribeStackResources returns ALL stack resources
• DescribeStackResource only returns information on a single resource and is the only source of the "Metadata"

I'm going to contact the Cloud Formation team and see if I can get some guidance on the desired API usage patterns. My best guess is that they would prefer general usage to call ListStackResources as it is pageable, and ensures accounts with large number of resources can get results quickly without downloading large responses.

What does this mean for the resource implementation? I'm not sure. There are few options:

• Model StackResource and StackResource summary separably. This ensures you can always hold onto the data returned without making n+1 requests
• Always use DescribeStackResources which seems to return nearly all of the data all of the time. Possible not a good idea for accounts with large number of resources. The metadata could be exposed as a separate action.
• Come up with a partial hydration story. This may require not exposing the "data" object on a resource except through the attribute getters.

I'll do some asking around, and come back with some more suggestions tomorrow.

[ktheory]

@trevorrowe:

The 2nd error requires that resources do not prefix their identifiers with their name. This avoids methods such as Event#event_id. Instead you would have Event#id.

Fixed in f6f21ae.

LMK what the CloudFormation team says about StackResources/Details/Summaries. Thanks again for the fast, thorough replies.

[trevorrowe]

I spoke with a member of the Cloud Formation team and have received some helpful feedback. The major take-away is that we should use ListStackResources when enumerating stack resources, and not DescribeStackResources. The primary reason is that the list call is paginated. This prevents client timeout issues when attempting to describe a large number of stack resources.

Given that I would like to expose two resource classes for a stack resource:

• StackResourceSummary
• StackResource

The StackResourceSummary would be enumerable (hasMany) from a Stack and would call the ListStackResource operation. It would use the shape by the same name and would not have a load operation. You could only fetch summaries in a loop like this:

stack.resource_summaries.each do |resource_summary|
  resource_summary.logical_id
  resource_summary.physical_id
  resource_summary.resource_type
  resource_summary.last_updated_timestamp
  resource_summary.resource_status
  resource_summary.resource_status_reason
end

The StackResource class would instead use the shape "StackResourceDetail" and would load itself by calling DescribeStackResource. This object would not be enumerable but would provide access to the description and metadata attributes of a stack resource.

We could use "subResource" on "Stack" to access "StackResource" objects:

resource = stack.resource('logical-resource-id')

# makes a single API call to DescribeStackResource to satisfy all of the
# following attribute methods
resource.stack_name
resource.logical_resource_id
resource.physical_resource_id
resource.last_updated_timestamp
resource.resource_status
resource.resource_status_reason
resource.description
resource.metadata

Lastly, we can add a hasOne association from StackResourceSummary to StackResource. This reference would allow you to go from summaries to their detail objects. This would be really useful if you need to collect metadata on resources.

# get metadata or description from a resource summary
summary = stack.resource_summaries.first

# get a resource from a resource summary
resource = summary.resource
resource.metadata
resource.description

# collect metadata on every stack resource into a hash
stack.resource_summaries.inject({}) do |metadata, summary|
  metadata[summary.logic_resource_id] = summary.resource.metadata
  metadata
end

Let me know if this makes sense or if you have any questions.

[trevorrowe]

I spoke with a member of the Cloud Formation team and have received some helpful feedback. The major take-away is that we should use ListStackResources when enumerating stack resources, and not DescribeStackResources. The primary reason is that the list call is paginated. This prevents client timeout issues when attempting to describe a large number of stack resources.

Given that I would like to expose two resource classes for a stack resource:

• StackResourceSummary
• StackResource

The StackResourceSummary would be enumerable (hasMany) from a Stack and would call the ListStackResource operation. It would use the shape by the same name and would not have a load operation. You could only fetch summaries in a loop like this:

stack.resource_summaries.each do |resource_summary|
  resource_summary.logical_id
  resource_summary.physical_id
  resource_summary.resource_type
  resource_summary.last_updated_timestamp
  resource_summary.resource_status
  resource_summary.resource_status_reason
end

The StackResource class would instead use the shape "StackResourceDetail" and would load itself by calling DescribeStackResource. This object would not be enumerable but would provide access to the description and metadata attributes of a stack resource.

We could use "subResource" on "Stack" to access "StackResource" objects:

resource = stack.resource('logical-resource-id')

# makes a single API call to DescribeStackResource to satisfy all of the
# following attribute methods
resource.stack_name
resource.logical_resource_id
resource.physical_resource_id
resource.last_updated_timestamp
resource.resource_status
resource.resource_status_reason
resource.description
resource.metadata

Lastly, we can add a hasOne association from StackResourceSummary to StackResource. This reference would allow you to go from summaries to their detail objects. This would be really useful if you need to collect metadata on resources.

# get metadata or description from a resource summary
summary = stack.resource_summaries.first

# get a resource from a resource summary
resource = summary.resource
resource.metadata
resource.description

# collect metadata on every stack resource into a hash
stack.resource_summaries.inject({}) do |metadata, summary|
  metadata[summary.logic_resource_id] = summary.resource.metadata
  metadata
end

Let me know if this makes sense or if you have any questions.

[ktheory]

Cool. I plan to try to implement that this weekend.

[ktheory]

@trevorrowe: Updated per your feedback. 0f0eafc supports the following:

stack = Aws::CloudFormation::Resource.new.stacks.first

stack.resource_summaries.to_a # ResourceSummaries using ListStackResources
stack.resource_summaries.map{|s| s.resource } # StackResourceDetail using DescribeStackResource
stack.resource(logical_id) # StackResourceDetail as subResource

👯 👯 👯

[coveralls]

Coverage increased (+0.44%) when pulling 0f0eafc on ktheory:cloudformation_resources into 9fadb05 on aws:master.

[ktheory]

👯