RDS cluster dynamic inventory support

[ryansb]

ISSUE TYPE

• Feature Pull Request

ANSIBLE VERSION

ansible 2.2.0 (rds-cluster-inventory-aioue 59e499f8f0) last updated 2016/07/06 10:57:31 (GMT -400)
  lib/ansible/modules/core: (detached HEAD 4a0a9cd1fc) last updated 2016/07/06 10:57:40 (GMT -400)
  lib/ansible/modules/extras: (detached HEAD e0b3e2f790) last updated 2016/07/05 15:58:33 (GMT -400)
  config file = 
  configured module search path = Default w/o overrides

SUMMARY

This is a rebased version of #16203 from @aioue, with the addition of a module_utils method for using boto3 connections in the dynamic inventory.

Example inventory output:

    {
      "db_clusters": {
        "ryansb-cluster-test": {
          "AllocatedStorage": 1,
          "AvailabilityZones": [
            "us-west-2a",
            "us-west-2b",
            "us-west-2c"
          ],
          "BackupRetentionPeriod": 1,
          "DBClusterIdentifier": "ryansb-cluster-test",
          "DBClusterMembers": [
            {
              "DBClusterParameterGroupStatus": "in-sync",
              "DBInstanceIdentifier": "ryansb-test",
              "IsClusterWriter": true,
              "PromotionTier": 1
            },
            {
              "DBClusterParameterGroupStatus": "in-sync",
              "DBInstanceIdentifier": "ryansb-test-us-west-2b",
              "IsClusterWriter": false,
              "PromotionTier": 1
            }
          ],
          "DBClusterParameterGroup": "default.aurora5.6",
          "DBSubnetGroup": "default",
          "DatabaseName": "mydb",
          "DbClusterResourceId": "cluster-OB6H7JQURFKFD4BYNHG5HSRLBA",
          "Endpoint": "ryansb-cluster-test.cluster-c9ntgaejgqln.us-west-2.rds.amazonaws.com",
          "Engine": "aurora",
          "EngineVersion": "5.6.10a",
          "MasterUsername": "admin",
          "Port": 3306,
          "PreferredBackupWindow": "06:09-06:39",
          "PreferredMaintenanceWindow": "mon:11:22-mon:11:52",
          "ReadReplicaIdentifiers": [],
          "Status": "available",
          "StorageEncrypted": false,
          "VpcSecurityGroups": [
            {
              "Status": "active",
              "VpcSecurityGroupId": "sg-47eaea20"
            }
          ]
        }
      },
      "rds": [
        "ryansb_test_c9ntgaejgqln_us_west_2_rds_amazonaws_com",
        "ryansb_test_us_west_2b_c9ntgaejgqln_us_west_2_rds_amazonaws_com"
      ],
      "rds_aurora": [
        "ryansb_test_c9ntgaejgqln_us_west_2_rds_amazonaws_com",
        "ryansb_test_us_west_2b_c9ntgaejgqln_us_west_2_rds_amazonaws_com"
      ],
      "rds_parameter_group_default_aurora5_6": [
        "ryansb_test_c9ntgaejgqln_us_west_2_rds_amazonaws_com",
        "ryansb_test_us_west_2b_c9ntgaejgqln_us_west_2_rds_amazonaws_com"
      ],
      "ryansb-test": [
        "ryansb_test_c9ntgaejgqln_us_west_2_rds_amazonaws_com"
      ],
      "ryansb-test-us-west-2b": [
        "ryansb_test_us_west_2b_c9ntgaejgqln_us_west_2_rds_amazonaws_com"
      ],
      "type_db_r3_large": [
        "ryansb_test_c9ntgaejgqln_us_west_2_rds_amazonaws_com",
        "ryansb_test_us_west_2b_c9ntgaejgqln_us_west_2_rds_amazonaws_com"
      ],
      "us-west-2": [
        "ryansb_test_c9ntgaejgqln_us_west_2_rds_amazonaws_com",
        "ryansb_test_us_west_2b_c9ntgaejgqln_us_west_2_rds_amazonaws_com"
      ],
      "us-west-2a": [
        "ryansb_test_c9ntgaejgqln_us_west_2_rds_amazonaws_com"
      ],
      "us-west-2b": [
        "ryansb_test_us_west_2b_c9ntgaejgqln_us_west_2_rds_amazonaws_com"
      ],
      "vpc_id_vpc_3ca34459": [
        "ryansb_test_c9ntgaejgqln_us_west_2_rds_amazonaws_com",
        "ryansb_test_us_west_2b_c9ntgaejgqln_us_west_2_rds_amazonaws_com"
      ]
    }

[tima]

• @defionscode

[aioue]

@ryansb no tags in your output? I pulled the tags for the cluster and inserted them into JSON before. You may not have tagged that cluster I guess but since code is same. 👍

[juliedavila]

My only slight concern with this is that this would be the first time, afaik, that the ec2.py script actually depends on ansible being installed on the executing system. At least as far as ec2.py goes, it has never had any dependency on Ansible. If the core/community team doesnt care, I dont really either. Just thought I'd point that out.

[ryansb]

@defionscode The impact for this seems pretty minimal, and in the future we can deprecate the ec2.py specific connection functions and share the ones in module_utils to reduce duplication. That'd be something I'd look to do in a future PR, not as part of this change.

@aioue I didn't make any changes to your code - here's the ec2.py diff between this PR and yours (the max_filter_value stuff is unrelated, and is from a commit I rebased on top of):

diff --git a/contrib/inventory/ec2.py b/contrib/inventory/ec2.py
index c9befc2..a870b8f 100755
--- a/contrib/inventory/ec2.py
+++ b/contrib/inventory/ec2.py
@@ -131,6 +131,8 @@ from boto import elasticache
 from boto import route53
 import six

+from ansible.module_utils import ec2 as ec2_utils
+
 HAS_BOTO3 = False
 try:
     import boto3
@@ -542,7 +544,12 @@ class Ec2Inventory(object):
             instance_ids = []
             for reservation in reservations:
                 instance_ids.extend([instance.id for instance in reservation.instances])
-            tags = conn.get_all_tags(filters={'resource-type': 'instance', 'resource-id': instance_ids})
+
+            max_filter_value = 199
+            tags = []
+            for i in range(0, len(instance_ids), max_filter_value):
+                tags.extend(conn.get_all_tags(filters={'resource-type': 'instance', 'resource-id': instance_ids[i:i+max_filter_value]}))
+
             tags_by_instance_id = defaultdict(dict)
             for tag in tags:
                 tags_by_instance_id[tag.res_id][tag.name] = tag.value
@@ -586,9 +593,10 @@ class Ec2Inventory(object):

     def include_rds_clusters_by_region(self, region):
         if not HAS_BOTO3:
-            module.fail_json(message="This module requires boto3 be installed - please install boto3 and try again")
-        
-        client = self.connect_to_aws(rds, region)
+            self.fail_with_error("Working with RDS clusters requires boto3 - please install boto3 and try again",
+                                 "getting RDS clusters")
+
+        client = ec2_utils.boto3_inventory_conn('client', 'rds', region, **self.credentials)
         clusters = client.describe_db_clusters()["DBClusters"]
         account_id = boto.connect_iam().get_user().arn.split(':')[4]
         c_dict = {}

I didn't add any tags to my test cluster, so that'd explain why my output lacks them.

[juliedavila]

Last thing, it would prob be good for it to handle situations in which results are paginated (though it's not too likely for there to be users with 100+ clusters...you never know)

[juliedavila]

shipit