New module cloudformation_facts #2329

jmenga · 2016-05-27T15:01:07Z

ISSUE TYPE

New Module Pull Request

COMPONENT NAME

cloudformation_facts

ANSIBLE VERSION

<!--- Paste verbatim output from “ansible --version” between quotes -->
ansible 2.1.0.0
  config file =
  configured module search path = Default w/o overrides

SUMMARY

Provides facts about AWS CloudFormation stacks. Note there is another open pull request #1230 however my module has much more functionality.

In its most basic form, it will return stack description and also provides some useful transformations (stack_outputs and stack_parameters in the example below) to make stack outputs and stack parameters easier to work with:

{
        "changed": false,
        "stack_description": {
            "CreationTime": "2016-05-27T02:02:23.997000+00:00",
            "DisableRollback": false,
            "NotificationARNs": [],
            "Parameters": [
                {
                    "ParameterKey": "SomeParameter",
                    "ParameterValue": "sg-abcd1234"
                },
                {
                    "ParameterKey": "DBInstanceClass",
                    "ParameterValue": "db.t2.medium"
                },
                {
                    "ParameterKey": "myVPC",
                    "ParameterValue": "vpc-abcd1234"
                }
            ],
            "Outputs": [
                {
                    "Description": "Stack Name",
                    "OutputKey": "StackName",
                    "OutputValue": "demo-stack"
                },
                {
                    "Description": "Some Output",
                    "OutputKey": "SomeOutput",
                    "OutputValue": "some-output-here"
                }
            ],
            "StackId": "arn:aws:cloudformation:ap-southeast-2:12345678:stack/demo-stack/0dcd0dd0-23af-11e6-bb66-50fae957fce6",
            "StackName": "demo-stack",
            "StackStatus": "CREATE_COMPLETE",
            "Tags": []
        },
        "stack_outputs": {
            "StackName": "demo-stack",
            "SomeOutput": "some-output-here",
        },
        "stack_parameters": {
            "SomeParameter": "sg-abcd1234",
            "DBInstanceClass": "db.t2.medium",
            "myVPC": "vpc-abcd1234"
        }
    }
}

A number of optional parameters can be specified to include the following additional stack information:

Stack Policy
Stack Template
Stack Events
Stack Resources

Example Usage:

# Output basic facts
- cloudformation_facts:
    stack_name: demo-stack

# Output all facts
- cloudformation_facts:
    stack_name: demo-stack
    all_facts: true

# Output basic facts + stack resources + stack template
- cloudformation_facts:
    stack_name: demo-stack
    stack_resources: true
    stack_template: true

Example expanded outputs:

{
        "changed": false,
        "stack_description": {
            "CreationTime": "2016-05-27T02:02:23.997000+00:00",
            "DisableRollback": false,
            "NotificationARNs": [],
            "Parameters": [
                {
                    "ParameterKey": "SomeParameter",
                    "ParameterValue": "sg-abcd1234"
                },
                {
                    "ParameterKey": "DBInstanceClass",
                    "ParameterValue": "db.t2.medium"
                },
                {
                    "ParameterKey": "myVPC",
                    "ParameterValue": "vpc-abcd1234"
                }
            ],
            "Outputs": [
                {
                    "Description": "Stack Name",
                    "OutputKey": "StackName",
                    "OutputValue": "demo-stack"
                },
                {
                    "Description": "Some Output",
                    "OutputKey": "SomeOutput",
                    "OutputValue": "some-output-here"
                }
            ],
            "StackId": "arn:aws:cloudformation:ap-southeast-2:12345678:stack/demo-stack/0dcd0dd0-23af-11e6-bb66-50fae957fce6",
            "StackName": "demo-stack",
            "StackStatus": "CREATE_COMPLETE",
            "Tags": []
        },
        # This is a dictionary mapping OutputKey values to OutputValue values
        "stack_outputs": {
            "StackName": "demo-stack",
            "SomeOutput": "some-output-here",
        },
        # This is a dictionary mapping ParameterKey values to ParameterValue values
        "stack_parameters": {
            "SomeParameter": "sg-abcd1234",
            "DBInstanceClass": "db.t2.medium",
            "myVPC": "vpc-abcd1234"
        },
       "stack_events": [
            {
                "EventId": "cef9c330-23b0-11e6-9836-50fae957fc4a",
                "LogicalResourceId": "my-rds",
                "PhysicalResourceId": "arn:aws:cloudformation:ap-southeast-2:12345678:stack/demo-stack/0dcd0dd0-23af-11e6-bb66-50fae957fce6",
                "ResourceStatus": "CREATE_COMPLETE",
                "ResourceType": "AWS::CloudFormation::Stack",
                "StackId": "arn:aws:cloudformation:ap-southeast-2:12345678:stack/demo-stack/0dcd0dd0-23af-11e6-bb66-50fae957fce6",
                "StackName": "demo-stack",
                "Timestamp": "2016-05-27T02:14:57.439000+00:00"
            },
           ....
        ],
       "stack_policy": {},
       "stack_resource_list": [
            {
                "LogicalResourceId": "DBinstance",
                "PhysicalResourceId": "my-rds",
                "ResourceStatus": "CREATE_COMPLETE",
                "ResourceType": "AWS::RDS::DBInstance",
                "StackId": "arn:aws:cloudformation:ap-southeast-2:12345678:stack/demo-stack/0dcd0dd0-23af-11e6-bb66-50fae957fce6",
                "StackName": "demo-stack",
                "Timestamp": "2016-05-27T02:14:55.879000+00:00"
            },
            ...
       ],
       # This is a dictionary mapping LogicalResourceID values to PhysicalResourceID values
       "stack_resources": {
            "DBinstance": "my-rds",
            ...
       },
       "stack_template": {
            "AWSTemplateFormatVersion": "2010-09-09",
            "Parameters": {
                ....
            },
            "Resources": {
                ....
            },
           ...  
     }
}

gregdek · 2016-05-27T16:25:10Z

Thanks @jmenga for this PR. This PR requires revisions, either because it fails to build or by reviewer request. Please make the suggested revisions. When you are done, please comment with text 'ready_for_review' and we will put this PR back into review.

[This message brought to you by your friendly Ansibull-bot.]

jmenga · 2016-05-27T20:26:28Z

ready_for_review

gregdek · 2016-05-27T22:25:14Z

Thanks @jmenga for this new module. When this module receives 'shipit' comments from two community members and any 'needs_revision' comments have been resolved, we will mark for inclusion.

[This message brought to you by your friendly Ansibull-bot.]

viper233 · 2016-05-31T21:50:40Z

@mixja I don't have ~/.boto set up, only AWS_REGION set in my environment.

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: botocore.exceptions.NoRegionError: You must specify a region.
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_K7IZmy/ansible_module_cloudformation_facts.py\", line 264, in <module>\n    main()\n  File \"/tmp/ansible_K7IZmy/ansible_module_cloudformation_facts.py\", line 234, in main\n    service_mgr = CloudFormationServiceManager(module)\n  File \"/tmp/ansible_K7IZmy/ansible_module_cloudformation_facts.py\", line 150, in __init__\n    self.client = boto3.client('cloudformation')\n  File \"/home/stephen/.local/lib/python2.7/site-packages/boto3/__init__.py\", line 79, in client\n    return _get_default_session().client(*args, **kwargs)\n  File \"/home/stephen/.local/lib/python2.7/site-packages/boto3/session.py\", line 200, in client\n    aws_session_token=aws_session_token, config=config)\n  File \"/usr/local/lib/python2.7/dist-packages/botocore/session.py\", line 796, in create_client\n    client_config=config, api_version=api_version)\n  File \"/usr/local/lib/python2.7/dist-packages/botocore/client.py\", line 62, in create_client\n    verify, credentials, scoped_config, client_config)\n  File \"/usr/local/lib/python2.7/dist-packages/botocore/client.py\", line 227, in _get_client_args\n    endpoint_url)\n  File \"/usr/local/lib/python2.7/dist-packages/botocore/client.py\", line 117, in _get_signature_version_and_region\n    service_model.endpoint_prefix, region_name, scheme=scheme)\n  File \"/usr/local/lib/python2.7/dist-packages/botocore/regions.py\", line 70, in construct_endpoint\n    raise NoRegionError()\nbotocore.exceptions.NoRegionError: You must specify a region.\n", "module_stdout": "", "msg": "MODULE FAILURE", "parsed": false}

I had to add the region to the client call

self.client = boto3.client('cloudformation', region_name='us-west-2')

I saw a commit with these details was removed. I noticed these modules were set up the same way

ansible/modules/extras/cloud/amazon/ecs_cluster.py
ansible/modules/extras/cloud/amazon/ecs_service_facts.py
ansible/modules/extras/cloud/amazon/ecs_service.py

though I haven't tried using them.

$ ansible --version
ansible 2.2.0 (devel 844b415066) last updated 2016/05/31 14:35:26 (GMT -700)

mixja · 2016-05-31T21:55:33Z

@viper233 - as per Boto3 docs you can use AWS_DEFAULT_REGION (AWS_REGION is not listed as a support variable) to set the region.

I will update the documentation of the module to reflect this needs to be set.

EDIT: updated incorrect Boto3 link above

viper233 · 2016-05-31T22:04:43Z

I pass the region in as a parameter.

# testing.yml
---
- hosts: localhost
  connection: local
  name: get a cloudformation stack facts
  gather_facts: False

  vars:
    - aws_region: "{{ lookup('env', 'AWS_REGION') }}"
    - aws_access_key: "{{ lookup('env', 'AWS_ACCESS_KEY_ID') }}"
    - aws_secret_key: "{{ lookup('env', 'AWS_SECRET_ACCESS_KEY') }}"

  tasks:
    - name: Print AWS region
      debug:
        var: aws_region

    - name: Get the cloudstack facts
      cloudformation_facts:
        stack_name: my_stack
        aws_access_key: "{{ aws_access_key }}"
        aws_secret_key: "{{ aws_secret_key }}"
        region: "{{ aws_region }}"
        stack_resources: true
      register: vpc

mixja · 2016-05-31T22:10:23Z

@viper233 - OK I see the problem, I will revise the module to support this style of configuration

viper233 · 2016-05-31T22:13:31Z

Is this also the case for the other modules too? I can try and test them later today and submit a PR

mixja · 2016-06-09T02:27:27Z

@viper233 - sorry not sure which other modules you are referring to. I have added support for specifying AWS connection/region parameters explicitly.

viper233 · 2016-06-09T03:08:51Z

I did a grep on the modules directories to see if there were any other modules using the boto3.client call, these were the others that I came up with.

ansible/modules/extras/cloud/amazon/ecs_cluster.py
ansible/modules/extras/cloud/amazon/ecs_service_facts.py
ansible/modules/extras/cloud/amazon/ecs_service.py

I haven't tested these modules yet. It's something I should do in my spare time.

Fluro · 2016-07-05T23:13:57Z

shipit

simonbrady · 2016-07-05T23:37:16Z

shipit

mixja · 2016-08-19T07:08:03Z

@gregdek - can you reattempt the Travis CI build as from the logs it looks like a build issue unrelated to the module code.

gregdek · 2016-08-19T13:23:03Z

@jmenga this is likely due to the age of the PR; please rebase against the latest devel. When you've done so, ping me and we'll put this module into the new module review meeting, since it has the necessary shipits.

mixja · 2016-08-19T22:02:15Z

Thanks @gregdek - rebased the PR and all checks pass

abadger · 2016-08-24T18:30:01Z

cloud/amazon/cloudformation_facts.py

+description:
+  - Gets information about an AWS CloudFormation stack
+dependencies:
+  - boto3>=1.0.0


This documentation field needs to be requirements, not dependencies.

I think that boto3 requires python >= 2.6? If so, need to mention python >= 2.6 as a requirement as well.

abadger · 2016-08-24T18:31:52Z

cloud/amazon/cloudformation_facts.py

+
+# import module snippets
+from ansible.module_utils.basic import *
+from ansible.module_utils.ec2 import *


Note -- we aren't requiring it yet but you don't have to treat these specially anymore. They can go at the top of the file with the other imports and you can import the specific things you need instead of doing a wildcard import.

abadger · 2016-08-24T18:38:53Z

needs_revision

gregdek · 2016-08-24T22:26:49Z

Thanks @jmenga for this PR. This PR requires revisions, either because it fails to build or by reviewer request. Please make the suggested revisions. When you are done, please comment with text 'ready_for_review' and we will put this PR back into review.

[This message brought to you by your friendly Ansibull-bot.]

mixja · 2016-08-26T01:54:10Z

ready_for_review

ryansb · 2016-08-30T14:20:40Z

cloud/amazon/cloudformation_facts.py

+            response = self.client.get_template(StackName=stack_name)
+            return response.get('TemplateBody')
+        except Exception as e:
+            self.module.fail_json(msg="Error getting stack template - " + str(e))


For all these fails, can you include the traceback like so:

try: # some stuff except Exception as e: self.module.fail_json(msg="Error getting stack template - " + str(e), exception=traceback.format_exc(e))

And import traceback at the top of the file?

ryansb · 2016-08-30T14:21:16Z

Other than the exception handling, this looks good.

jmenga · 2016-09-06T02:42:08Z

ready_for_review

gregdek · 2016-09-06T04:26:22Z

Thanks @jmenga for this new module. When this module receives 'shipit' comments from two community members and any 'needs_revision' comments have been resolved, we will mark for inclusion.

[This message brought to you by your friendly Ansibull-bot.]

ryansb · 2016-09-06T21:13:39Z

Looks great - I'm going to commit a fix to one of the boto3 connection creation manually. There is a problem with using this module with profiles.

gregdek added the cloud label May 27, 2016

gregdek added new_plugin needs_revision labels May 27, 2016

gregdek added community_review and removed needs_revision labels May 27, 2016

mixja added 7 commits August 20, 2016 09:57

Add cloudformation_facts module

1950178

Update docs to indicated stack name or ID can be used

4ad0838

Add boto3 dependency info, remove unnecessary AWS connection info check

3d5aa01

Move example outputs to EXAMPLES section

c1b79ec

Update for Python 3.x compatibility

cb526c5

Add support for standard AWS connection parameters

03c2ce8

Use list_stack_resources for stacks with > 100 resources

d6c8e81

jmenga force-pushed the cloudformation_facts branch from acc6067 to d6c8e81 Compare August 19, 2016 21:58

gregdek mentioned this pull request Aug 20, 2016

Standing Agenda, New Modules Meeting ansible/community#92

Closed

abadger reviewed Aug 24, 2016
View reviewed changes

gregdek added needs_revision and removed community_review labels Aug 24, 2016

Add correct facts support

94bd40d

Clean up imports, update docs

7a6ba16

ryansb reviewed Aug 30, 2016
View reviewed changes

jmenga added 2 commits September 6, 2016 14:31

Update exception formatting

572bd9a

Remove boto exception handler

6480e72

gregdek added community_review and removed needs_revision labels Sep 6, 2016

ryansb merged commit 1f79c35 into ansible:devel Sep 6, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New module cloudformation_facts #2329

New module cloudformation_facts #2329

jmenga commented May 27, 2016

gregdek commented May 27, 2016

jmenga commented May 27, 2016

gregdek commented May 27, 2016

viper233 commented May 31, 2016

mixja commented May 31, 2016 •

edited

Loading

viper233 commented May 31, 2016

mixja commented May 31, 2016

viper233 commented May 31, 2016

mixja commented Jun 9, 2016

viper233 commented Jun 9, 2016

Fluro commented Jul 5, 2016

simonbrady commented Jul 5, 2016

mixja commented Aug 19, 2016

gregdek commented Aug 19, 2016

mixja commented Aug 19, 2016

abadger Aug 24, 2016

abadger Aug 24, 2016

abadger commented Aug 24, 2016

gregdek commented Aug 24, 2016

mixja commented Aug 26, 2016

ryansb Aug 30, 2016

ryansb commented Aug 30, 2016

jmenga commented Sep 6, 2016

gregdek commented Sep 6, 2016

ryansb commented Sep 6, 2016

New module cloudformation_facts #2329

New module cloudformation_facts #2329

Conversation

jmenga commented May 27, 2016

ISSUE TYPE

COMPONENT NAME

ANSIBLE VERSION

SUMMARY

gregdek commented May 27, 2016

jmenga commented May 27, 2016

gregdek commented May 27, 2016

viper233 commented May 31, 2016

mixja commented May 31, 2016 • edited Loading

viper233 commented May 31, 2016

mixja commented May 31, 2016

viper233 commented May 31, 2016

mixja commented Jun 9, 2016

viper233 commented Jun 9, 2016

Fluro commented Jul 5, 2016

simonbrady commented Jul 5, 2016

mixja commented Aug 19, 2016

gregdek commented Aug 19, 2016

mixja commented Aug 19, 2016

abadger Aug 24, 2016

Choose a reason for hiding this comment

abadger Aug 24, 2016

Choose a reason for hiding this comment

abadger commented Aug 24, 2016

gregdek commented Aug 24, 2016

mixja commented Aug 26, 2016

ryansb Aug 30, 2016

Choose a reason for hiding this comment

ryansb commented Aug 30, 2016

jmenga commented Sep 6, 2016

gregdek commented Sep 6, 2016

ryansb commented Sep 6, 2016

mixja commented May 31, 2016 •

edited

Loading