How can AWS CloudFormation help?

[cdandekar]

Troposphere contributors and users,

Is there anything the AWS CloudFormation service could provide that can help in using and enhancing troposphere? (For example, any additional API, any update to the template format, etc.)

Sincerely,
The AWS CloudFormation Team

[robzienert]

Hi Chetan,

This isn't Troposphere-specific, but I'd really like a new intrinsic function added to the effect of GetSubnets(Network=private), where "Network" is a Tag key on the subnet and "private" is the Tag value. It would be super helpful in creating ELBs & ASGs that belong to specific subnets without having to pass them in as a list Parameter.

[cdandekar]

@robzienert, Thank you for the feedback.

[markpeek]

Hi Chetan,
Good to hear from you and the CloudFormation team. If I remember the last we talked was re:invent 2013 with @bobveznat and @phobologic . We have definitely seen an uptick on Troposphere users with quite a few people contributing patches to it. The project started out as an alternative to writing json and to allow Python constructs (comments, loops, etc.) in creating the templates. It evolved into doing more object validation to get any easy issues caught at template creation time instead of template execution time. This has proven to be very useful to us and our users.

In thinking of ways the CloudFormation team and service could help us I've tried to bucket it into some major themes:

Awareness

• To be honest I'm not sure how users find out about troposphere but I have seen blog posts, slide presentations and other social media commenting about it. It would be nice to have this project listed on the AWS 3rd party library page. I had submitted a form at one point but never heard back on it.
• While the CloudFormation team presents at AWS and other venues the json templates, it might be nice to give a list of 3rd party utilities or use them as examples as well.

Keep updated

• New CloudFormation release show up out of the blue and we then scramble to figure out what changed. It also seems like docs get updated without notifications. The release notes help but still require some sleuthing to know exactly what changes. It would be nice to have a more explicit list of things changed from release to release, or updates (diffs) to the docs so we can update our code.
• If changes are coming we could coordinate and assist in integrating any code changes to Troposphere should the CloudFormation team wish to help us.

Accuracy and Thoroughness

• We have been doing our best to include as much of the AWS resources as possible and to add accurate validation. However, we don't have a way to cross check exhaustively against the docs. It would be nice to have some form of machine readable format to check resource types, property names and required against.
• Also, it might be good for the CF team to code review Troposphere for inaccuracies.

Any help the CloudFormation team wants to give us would be appreciated. This could be giving us advance warning on changes, contributing to the project and/or helping to promote it.

Thanks,
Mark, Mike and Bob

[cdandekar]

Thank you @markpeek @bobveznat @phobologic, we will loook into this.

[cdandekar]

The release notes help but still require some sleuthing to know exactly what changes. It would be nice to have a more explicit list of things changed from release to release, or updates (diffs) to the docs so we can update our code.

Can you elaborate on / provide an example of what the release notes leave to be desired?

[markpeek]

As an example, let's take the most recent release notes here which states:

AWS CloudFormation added two new properties for Amazon RDS database instances. You can now associate an option group with a database instance and specify the database instance storage type. For more information, see AWS::RDS::DBInstance.

Following the link included goes here in which we then need to figure out which properties were actually added. A good guess is the "OptionGroupName" and "StorageType" properties.

A couple of points:

1. It would be nice if the actual properties names were mentioned (sometimes they are and sometimes not)
2. Better would be to link the property names directly from the release notes to the docs via html anchor tags
3. Having a change log in the user guide would allow people to know what changed and when
4. Having machine readable specifications would help by allowing an automated diff operation to find the changes

Some release note entries are more explicit than others, for example, the ELB ConnectionSettings in that same release note. Also, there are sometimes documentation updates that do not appear to be mentioned in the release notes which makes it difficult to track those changes.

I hope this helps to elaborate what we do to track the CloudFormation changes from the release notes.

[cdandekar]

Thanks for the additional information, Mark. If we were to provide a "machine readable specification" what kind of format would you like to see that in?

[markpeek]

That question is always an interesting debate. Right now the two main formats would be XML and JSON. I might lean towards JSON given the use of it in CloudFormation and the (perceived) trend from AWS to use more JSON than XML (for example, botocore/aws-cli being data driven from JSON).

[cdandekar]

Would a (json-schema.org) based CloudFormation schema be suitable for you or would you prefer a different JSON-based format for any reason?

[markpeek]

I had not intended to be prescriptive in what format to use since it is really up to AWS to define and maintain it. But yes, the (json-schema.org) based CloudFormation schema looks like a fine direction.

[cdandekar]

Appreciate you making that distinction between a requirement and a prescription, Mark :-). We listen to our customers and partners, so just wanted to hear your opinion on what kinds of format would work best for you.

[markpeek]

BTW, some resource types might be trickier to document than others. I was just looking at issue #197 and I'm seeing discrepancies in the docs. Some examples:

• The Engine property isn't needed for a read replica but it doesn't state that as a condition.
• AllocatedStorage has "Required: Yes" but again is not needed for a read replica.
  And there are quite a few constraints for what properties not to use when using SourceDBInstanceIdentifier.

Note: in some ways having a new resource type of AWS::RDS::DBInstanceReplica might make it easier to distinguish and specify the correct properties. In other words, then there could be less conditionals and more "required" fields.

[bedge]

AWS could provide troposphere source files for existing sample templates in addition to json.
This would help new users by having a ready made library set to use as examples. There is no simple path to go from existing sample templates to using troposphere unless one starts from scratch.
cfn2py is a great idea, but it lacks functionality and even if it could parse existing complex templates, does not provide what one could call optimized, elegant source files.

[phobologic]

Just another example where I'm sure documented changes would help so much: GH-210

I don't think we got ElasticBeanstalk totally wrong when it was accepted, so my guess is that things changed drastically at some point. Since nobody on the Cloudtools org uses ElasticBeanstalk much, we don't notice these changes.

[pas256]

I'm just gonna say this... part of the reason this project started was people hate (I don't use that word lightly) hand coding JSON. It is a format that can be written by humans, but is very much a machine format. YAML is significantly easier for a human to read and write and most of all, it supports comments. If anything were to increase the adoption of CloudFormation, accepting a template written in YAML would be it.

Other than that, simply acquire and integrate CloudCoreo or Luminal :-P
Both a the next iteration of CloudFormation.

[pas256]

:)

[domdivakaruni]

@markpeek @bobveznat @phobologic picking back up on this after a while of radio silence. We appreciate what you guys do. We're happy to help keep the channel open and help out in anyway. We're working on a schema BTW so stay tuned.

[phobologic]

Thanks @domdivakaruni !