Several features (AWS CloudFormation Support, IAM Support, Kinesis, DynamoDB, and local auth options) #172

Merged
merged 14 commits into from Mar 19, 2016

Conversation

Projects
None yet
5 participants
@vancluever
Contributor

vancluever commented Sep 13, 2015

Hey guys,

This PR adds several new things that I am intending to use to make my job easier with managing our AWS resources.

Local Credential and STS Support (Including MFA)

Local credentials (ie: via ~/.aws/credentials) are now supported, as is STS roles thru the role_arn parameter. Further to the latter, MFA is also supported via the mfa_serial parameter and a MFA code attribute. Usage scenarios are documented in the README (including an example of automation with chef-zero).

Region Resource Parameter

I needed to add the region resource parameter as well as it looks like if the cookbook was being run outside of EC2, I was getting issues where it would time out trying to auto-detect region. Possibly related to #160.

CloudFormation Support

Support for managing CloudFormation templates has been added - the template is stored in the consuming cookbook's files store, with parameters supplied much like in a Chef template resource. Resource supports creation, update, and delete, with the update part actually checking the template's JSON to ensure something has actually changed in the template before calling update_stack. disable_rollback and stack_policy_body are also there. Usage details are again, in the README.

IAM Support

Finally, support for creation of IAM users, groups, policies, and roles has been added as well. These were added as I was not entirely satisfied with how CloudFormation manages the naming of IAM resources (there still seems to be no way to control user names, for example). Full details of how to use these resources are in the README, there is quite a bit of detail, so for summary's sake I won't mention all of the features.

What has been tested?

I have tested most of the stuff using a local cookbook running chef-zero, including create/update/delete cycles for every resource here. There has been some refactoring done to ensure compliance with rubocop, so I will be doing another round of testing tonight, including testing the disable_rollback and stack_policy_body features of the cfn_stack resource which have not been tested yet. However, the PR is ready to at least get the ball going on so I thought I would submit it.

I have not re-tested the existing resources as I have not had cycles to do so, and assuming that the new parameters will not break any functionality as they only really come into play when setting up the session in libraries/ec2.rb.

Conclusion

If there's anything else I need to do to help get this PR accepted, please let me know!

Thanks,

--Chris

@vancluever

This comment has been minimized.

Show comment
Hide comment
@vancluever

vancluever Sep 15, 2015

Contributor

Guys,

Looks like the MFA stuff is probably a no-go - it makes sense when I try it out now, but trying to recycle the MFA code across multiple STS sessions does not work. I have removed the functionality, and instead put a section in the README with an example of how one can use MFA and supply the session data to a resource. Testing this now, but in theory it should work.

--Chris

Contributor

vancluever commented Sep 15, 2015

Guys,

Looks like the MFA stuff is probably a no-go - it makes sense when I try it out now, but trying to recycle the MFA code across multiple STS sessions does not work. I have removed the functionality, and instead put a section in the README with an example of how one can use MFA and supply the session data to a resource. Testing this now, but in theory it should work.

--Chris

@vancluever

This comment has been minimized.

Show comment
Hide comment
@vancluever

vancluever Sep 15, 2015

Contributor

New MFA style works. Also added an account_id attribute to aws_iam_policy as you cannot determine an ARN based off username when using non-user credentials.

Contributor

vancluever commented Sep 15, 2015

New MFA style works. Also added an account_id attribute to aws_iam_policy as you cannot determine an ARN based off username when using non-user credentials.

Chris Marchesi
AWS CloudFormation Support, IAM Support, and Additional Auth Options
 * Adding the aws_cfn_stack LWRP
 * Adding the aws_iam_user, aws_iam_group, aws_iam_policy, and aws_iam_role LWRPs
 * Adding global facilities to support using local credentials for auth (ie: ~/.aws/credentials)
@vancluever

This comment has been minimized.

Show comment
Hide comment
@vancluever

vancluever Sep 15, 2015

Contributor

OK, everything works (including the other outstanding CloudFormation bits mentioned at the beginning). I've re-based again. This should be it, but I will update if I notice anything else.

Contributor

vancluever commented Sep 15, 2015

OK, everything works (including the other outstanding CloudFormation bits mentioned at the beginning). I've re-based again. This should be it, but I will update if I notice anything else.

@vancluever vancluever changed the title from AWS CloudFormation Support, IAM Support, and Additional Auth Options to AWS CloudFormation Support, IAM Support, and Additional Auth Options (also Kinesis) Sep 18, 2015

@vancluever

This comment has been minimized.

Show comment
Hide comment
@vancluever

vancluever Sep 18, 2015

Contributor

One update - aws_kinesis_stream to manage Kinesis streams (another thing you can't control naming for in CloudFormation!)

Let me know if this should be squashed.

--Chris

Contributor

vancluever commented Sep 18, 2015

One update - aws_kinesis_stream to manage Kinesis streams (another thing you can't control naming for in CloudFormation!)

Let me know if this should be squashed.

--Chris

Chris Marchesi and others added some commits Oct 2, 2015

Chris Marchesi
Addition of the DyanmoDB resource, can add add tables, and also add g…
…lobal secondary indexes to existing tables.
Merge pull request #1 from paybyphone/dynamodb_resources
Addition of the DyanmoDB resource, can add add tables, and also add global secondary indexes to existing tables

@vancluever vancluever changed the title from AWS CloudFormation Support, IAM Support, and Additional Auth Options (also Kinesis) to Several features (AWS CloudFormation Support, IAM Support, Kinesis, DynamoDB, and local auth options) Oct 10, 2015

@vancluever

This comment has been minimized.

Show comment
Hide comment
@vancluever

vancluever Oct 10, 2015

Contributor

Guys,

Another feature - DynamoDB. The resource allows for creation of tables and also the ability to add global secondary indexes on the fly.

--Chris

Contributor

vancluever commented Oct 10, 2015

Guys,

Another feature - DynamoDB. The resource allows for creation of tables and also the ability to add global secondary indexes on the fly.

--Chris

Chris Marchesi
* New feature: aws_cfn_stack now supports IAM capability. See README…
… for details.

 * aws_cfn_stack: A small bugfix to ensure that files that are included in subdirectories of recipes get written out to cache correctly.
@vancluever

This comment has been minimized.

Show comment
Hide comment
@vancluever

vancluever Oct 21, 2015

Contributor

Added iam_capability to the aws_cfn_stack resource, and also a small bugfix on how files are cache for the resource that was coming up when the file resided in a subdirectory within the consuming cookbook (example: files/default/cloudformation/template.tpl).

PS: Any word on when this might be reviewed and/or merged? I see that our fork is now falling behind commit wise. I have another commit that I wanted to send along (some readability fixes on the Dynamo resource, mainly), but it's been awfully quiet here.

--Chris

Contributor

vancluever commented Oct 21, 2015

Added iam_capability to the aws_cfn_stack resource, and also a small bugfix on how files are cache for the resource that was coming up when the file resided in a subdirectory within the consuming cookbook (example: files/default/cloudformation/template.tpl).

PS: Any word on when this might be reviewed and/or merged? I see that our fork is now falling behind commit wise. I have another commit that I wanted to send along (some readability fixes on the Dynamo resource, mainly), but it's been awfully quiet here.

--Chris

Chris Marchesi and others added some commits Oct 29, 2015

Merge pull request #2 from paybyphone/develop
Fix for aws_cfn_stack running under chef-zero on Windows (no more template file stage)
Merge pull request #3 from paybyphone/develop
Fix to allow for template update on parameter change
DynamodDB cleanup
 * Refactoring the dynamodb_table resource provider
 * Fixtures for testing table creation and update
 * rspec tests in Kitchen to test Dynamo thru API
@mvillis

This comment has been minimized.

Show comment
Hide comment
@mvillis

mvillis Nov 27, 2015

would be great to get this PR over the line. CF support into the AWS cookbook would be amazing!

mvillis commented Nov 27, 2015

would be great to get this PR over the line. CF support into the AWS cookbook would be amazing!

@vancluever

This comment has been minimized.

Show comment
Hide comment
@vancluever

vancluever Jan 8, 2016

Contributor

Guys,

I've seen no traction on this PR, and the cookbook has been quiet since late November - any word on what's going on?

Would love to get this merged so that I can point some of our repos using this back to upstream.

Contributor

vancluever commented Jan 8, 2016

Guys,

I've seen no traction on this PR, and the cookbook has been quiet since late November - any word on what's going on?

Would love to get this merged so that I can point some of our repos using this back to upstream.

@tas50

This comment has been minimized.

Show comment
Hide comment
@tas50

tas50 Jan 22, 2016

Member

@vancluever Sorry about taking so long to get back to you on this PR. It's certainly a set of features we'd like to get into the cookbook. I'll carve out some time to take a look at things and get back to you.

Member

tas50 commented Jan 22, 2016

@vancluever Sorry about taking so long to get back to you on this PR. It's certainly a set of features we'd like to get into the cookbook. I'll carve out some time to take a look at things and get back to you.

@vancluever

This comment has been minimized.

Show comment
Hide comment
@vancluever

vancluever Jan 22, 2016

Contributor

Wooh! Thanks a ton @tas50! I think my Ruby's gotten a lot better since I initially sent this along, so if there's some refactoring that needs done, let me know and I can set some time aside to get it done. I'd imagine some squashing is also in order, but I'll wait for the review.

Contributor

vancluever commented Jan 22, 2016

Wooh! Thanks a ton @tas50! I think my Ruby's gotten a lot better since I initially sent this along, so if there's some refactoring that needs done, let me know and I can set some time aside to get it done. I'd imagine some squashing is also in order, but I'll wait for the review.

- aws_ssh_key_id: <%= ENV['AWS_KEYPAIR_NAME'] %>
- availability_zone: <%= ENV['AWS_AVAILABILITY_ZONE'] %>
+ iam_profile_name: ChefKitchenTestAws
+ aws_ssh_key_id: chef-kitchen-test-aws

This comment has been minimized.

@tas50

tas50 Mar 10, 2016

Member

This file needs to get updated with the contents of the latest file in master. I've updated all the boxes to the latest released and made sure we test on RHEL platforms as well. Also we need to make sure that any driver config is configurable and works for any users. We need to be able to set these in travis at some point so we can test on each PR.

@tas50

tas50 Mar 10, 2016

Member

This file needs to get updated with the contents of the latest file in master. I've updated all the boxes to the latest released and made sure we test on RHEL platforms as well. Also we need to make sure that any driver config is configurable and works for any users. We need to be able to set these in travis at some point so we can test on each PR.

@@ -436,6 +528,331 @@ Allows detailed CloudWatch monitoring to be enabled for the current instance.
aws_instance_monitoring "enable detailed monitoring"
+
+## aws_cfn_stack

This comment has been minimized.

@tas50

tas50 Mar 10, 2016

Member

can we call this aws_cloudformation_stack even though it's a bit more to type. It's much more clear when the user reads over cookbook code.

@tas50

tas50 Mar 10, 2016

Member

can we call this aws_cloudformation_stack even though it's a bit more to type. It's much more clear when the user reads over cookbook code.

@iennae

This comment has been minimized.

Show comment
Hide comment
@iennae

iennae Mar 11, 2016

Contributor

@vancluever We are really excited to merge these changes in, but we could really use your help in doing so. Could you rebase and let us know if some of the changes that have happened in the time since you wrote your PR work for your changes (like the query_aws_region function which should probably be refactored into a general library ).

Also please factor in @tas50 line item comments. Thank you so much for your contributions. We will get this merged in ASAP.

Contributor

iennae commented Mar 11, 2016

@vancluever We are really excited to merge these changes in, but we could really use your help in doing so. Could you rebase and let us know if some of the changes that have happened in the time since you wrote your PR work for your changes (like the query_aws_region function which should probably be refactored into a general library ).

Also please factor in @tas50 line item comments. Thank you so much for your contributions. We will get this merged in ASAP.

@vancluever

This comment has been minimized.

Show comment
Hide comment
@vancluever

vancluever Mar 16, 2016

Contributor

Hey @iennae and @tas50 - I just wanted to let you two know that I haven't forgotten about this and it's on my radar to do relatively shortly. Hopefully I have time in the next few days to perform the needful.

PS @tas50 - I can just drop the kitchen stuff from this PR, I don't necessarily want to go out of scope. I will see if I can keep the integration tests, and maybe just blacklist the .kitchen.cloud.yml file from the commit, unless you think that there's some value in keeping them.

Will reply more in the next couple of days!

Contributor

vancluever commented Mar 16, 2016

Hey @iennae and @tas50 - I just wanted to let you two know that I haven't forgotten about this and it's on my radar to do relatively shortly. Hopefully I have time in the next few days to perform the needful.

PS @tas50 - I can just drop the kitchen stuff from this PR, I don't necessarily want to go out of scope. I will see if I can keep the integration tests, and maybe just blacklist the .kitchen.cloud.yml file from the commit, unless you think that there's some value in keeping them.

Will reply more in the next couple of days!

@tas50

This comment has been minimized.

Show comment
Hide comment
@tas50

tas50 Mar 18, 2016

Member

@vancluever The tighter the scope the better. I really want to start ripping this cookbook apart, but I'd like to get this landed first so we don't continue to go back and forth on rebasing.

Member

tas50 commented Mar 18, 2016

@vancluever The tighter the scope the better. I really want to start ripping this cookbook apart, but I'd like to get this landed first so we don't continue to go back and forth on rebasing.

@tas50 tas50 merged commit f495672 into chef-cookbooks:master Mar 19, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment