CloudWanderer is made up of three core components.
- The cloud interface (e.g.
~cloudwanderer.aws_interface.CloudWandererAWSInterface
). Responsible for discovering the resources that exist in your cloud provider. - The storage connector (e.g.
~cloudwanderer.storage_connectors.DynamoDbConnector
). Responsible for storing the discovered resources in your storage mechanism of choice. - The
~cloudwanderer.cloud_wanderer.CloudWanderer
class. Responsible for bringing the interface and storage connectors together to make them easier to work with.
If you don't mind that your data is thrown away as soon as your python
executable stops you can test CloudWanderer using the Memory Storage Connector!
It's wise to do this in an interactive environment otherwise you may spend an inordinate amount of time re-querying your AWS environment!
DynamoDB has a Docker Image that allows you to run a local, persistent DynamoDB. This provides us with a cheap and easy way to start trying out CloudWanderer.
This starts a DynamoDB docker image on your local machine and tells it to persist data into the current directory in a shared database file shared-local-instance.db
. This allows the data to persist even if you stop the container.
This creates an alternative storage connector that points at your local DynamoDB
This passes the storage connector that points at your local DynamoDB into a new wanderer and now all subsequent CloudWanderer operations will occur against your local DynamoDB!
Writing all supported_resources
in all regions is as simple as using the ~cloudwanderer.cloud_wanderer.CloudWanderer.write_resources
method.
In that block we are:
- Creating a storage connector (in this case DynamoDB)
- Initialising the storage connector (in this case creating a dynamodb table called
cloud_wanderer
- Creating a wanderer and using
~cloudwanderer.cloud_wanderer.CloudWanderer.write_resources
to get all resources in all regions.
Important: This will create DynamoDB table in your AWS account and write a potentially large number of records to it which may incur some cost. See earlier examples for how to test against a local DynamoDB or memory.
Writing VPCs is as simple as passing the resource_types
argument.
Some resource types take a very long time to query (e.g. EC2 Images) and depending on what you're using your data for may not be worth the time.
If you're writing an event driven discovery mechanism it can be very useful to be able to update an individual resource without discovering all of the other resources of that type as well.
Warning
If the resource is not found it will delete it from your storage connector. This applies to both ~cloudwanderer.cloud_wanderer.CloudWanderer.write_resource
and ~cloudwanderer.cloud_wanderer.CloudWanderer.write_resources
.
You'll notice here we're calling a property urn
in order to print the region. URNs <reference/urn>
are CloudWanderer's way of uniquely identifying a resource.
You can also see we're printing the vpc's state
and is_default
attributes. It's very important to notice the ~cloudwanderer.cloud_wanderer_resource.CloudWandererResource.load
call beforehand which loads the resource's data. Resources returned from any ~cloudwanderer.storage_connectors.DynamoDbConnector.read_resources
call on ~cloudwanderer.storage_connectors.DynamoDbConnector
are lazily loaded unless you specify the urn=
argument. This is due to the sparsely populated global secondary indexes in the DynamoDB table schema.
Once you've called ~cloudwanderer.cloud_wanderer_resource.CloudWandererResource.load
you can access any property of the AWS resource that is returned by its describe method. E.g. for VPCs see boto3:EC2.Client.describe_vpcs
. These attributes are stored as snake_case instead of the APIs camelCase, so isDefault
becomes is_default
.
In CloudWanderer, a subresource is a resource which does not have its own unique identifier in the cloud provider. It depends upon its parent resource for its identity.
An example of a subresource is a AWS IAM Role Inline Policy. The Role has an ARN (AWS's unique identifier), but the policy does not. When interacting with the AWS API you can only retrieve an inline policy by specifyng the policy name and the role name/ARN. This makes it qualify as a subresource in CloudWanderer terminology.
This is unlike Boto3, where a subresource is any resource dependent on a parent resource (e.g. a subnet is a subresource of a VPC). A subnet does not fit the CloudWanderer definition of a subresource however, because a subnet has its own unique identifier and can therefore be retrieved from the API without specifying the VPC of which it is a part.
Let's say we want to get a list of role policies. We can start by getting the role
Next we need to find out what policies are attached, we can either do this with the secondary attributes.
Or we can do it with the ~cloudwanderer.cloud_wanderer_resource.CloudWandererResource.dependent_resource_urns
property.
Then we can lookup the inline policy
Some resources require additional API calls beyond the initial list
or describe
call to retrieve all their metadata. These are known as Secondary Attributes. These secondary attributes are written as part of ~cloudwanderer.cloud_wanderer.CloudWanderer.write_resources
.
Let's say we want to get the value of enableDnsSupport
for a VPC. We can get this by accessing the enable_dns_support
attribute on the VPC object.
CloudWanderer deletes resources which no longer exist automatically when you run: ~cloudwanderer.cloud_wanderer.CloudWanderer.write_resources
.
This has some complexity with regional resources that only exist via global APIs. For example S3 buckets are regional resources, but S3 is a global service so when you call ~cloudwanderer.cloud_wanderer.CloudWanderer.write_resources
for S3 buckets in us-east-1
you will get buckets from all regions due to the nature of the API.
This also means that you will delete S3 buckets that no longer exist from all regions when you call ~cloudwanderer.cloud_wanderer.CloudWanderer.write_resources
in us-east-1
.
Deleting individual resources (if necessary), can be done by calling ~cloudwanderer.storage_connectors.DynamoDbConnector.delete_resource
directly on the storage connector.
e.g.