Updated AWS Config worker for resource exclusions

- Added support for AWS Config resource exclusions

mkdocs/userGuide/awsConfig/AWSConfigWorker.md

@@ -1,14 +1,16 @@
 # AWS Config Worker Ship User Guide
 This is the main user guide for the AWS Config worker ship. This page documents all that there is to know about this worker ship.
 
+!!! warning "Schema Change June 2023"
+    The initial schema for this worker was changed in June 2023 to introduce support for AWS Config's resource
+    exclusion feature in the recorders. See the Payload Templates section for details on what the current schema is.
+
 ## What does it do?
 This is a very important worker ship as it allows you to easily enable [AWS Config](https://aws.amazon.com/config/) to all accounts and regions in your infrastructure. AWS Config will also be used with other workers to act as a cache of AWS resources. This is done to avoid API rate limiting when describing resources.
 
 ## Why did we make this?
 AWS Config is a very important visibility tool in AWS. It is also a very effective cache of resource state that can be queried without causing rate limits for the respective services. However, AWS Config recorder enablement is not simple to do without this tool. AWS Organizations lacks the ability to enable AWS Config recorders across your environment, and more importantly, you will likely need the ability to customize how AWS Config is configured throughout your accounts and regions.
-
 Being able to customize how AWS Config is enabled across your infrastructure is highly desireable. This is because AWS Config bills you for each configuration change that is recorded. If you have an AWS account and region with a lot of changes being made to a given resource type, then you may incur a lot of charges for those resources being recorded. In such a case, you may still want to enable it for the resource types that don't frequently change so you can still have reasonable configuration history coverage.
-
 With this worker, you can have it all, as you can define in one place how AWS Config should be configured, and you can specify account/region specific overrides. Starfleet will task the workers accordingly to implement the changes throughout very quickly. This will also do both, detect and correct any drift that appears with AWS Config recorders.
 
 ## What does it NOT do?

mkdocs/userGuide/awsConfig/Template.md

@@ -4,6 +4,15 @@ In this section, we discuss the AWS Config worker templates. The Config Worker i
 ## Template Schema
 The only required field is a very important one and that is the `DefaultConfiguration` section. Below is a sample template where this is defined:
 
+!!! warning "Schema Change June 2023"
+    The initial schema for this worker was changed in June 2023 to introduce support for AWS Config's resource
+    exclusion feature in the recorders. The original schema is no longer functional. The main change is in the `RecordingGroup` section, which is defined
+    in detail below.
+
+Below is the most realistic example. We are not going to record EC2 instances anywhere because in this sample environment, they change too much and are too costly to have recorded.
+Thus, by default, we are recording all resources except EC2 instances and global resources like IAM. For IAM and other global resources, we only want to record them in us-east-1
+to avoid duplicate resource recording and the associated costs with that.
+
 ```yaml
 TemplateName: AWSConfigEnablement
 TemplateDescription: Enabling AWS Config
@@ -20,11 +29,28 @@ DefaultConfiguration:
         ConfigRoleName: AWSConfigRole
         RecordingEnabled: True
         RecordingGroup:
-            ResourceTypes:
-                - ALL
-            GlobalsInRegions:
-                - us-east-1
+            RecordEverythingExcept:
+                - AWS::EC2::Instance
+                - AWS::IAM::Role    # Record IAM in us-east-1 only -- see below
+                - AWS::IAM::Group
+                - AWS::IAM::User
     RetentionPeriodInDays: 2557
+AccountOverrideConfigurations:
+    -
+        IncludeAccounts:
+            AllAccounts: True
+        IncludeRegions:
+            - us-east-1
+        DeliveryChannelDetails:
+            BucketName: your-s3-bucket-here
+            S3DeliveryFrequency: Twelve_Hours
+        RecorderConfiguration:
+            ConfigRoleName: AWSConfigRole
+            RecordingEnabled: True
+            RecordingGroup:
+                RecordEverythingExcept:  # IAM will only be in us-east-1 to avoid recording global resources as they would be duplicated and increase recording cost
+                    - AWS::EC2::Instance
+        RetentionPeriodInDays: 2557
 ```
 
 The `DefaultConfiguration` defines how AWS Config should be configured by default. That is to say, how it should be configured for all non-excluded (__and__ not overridden -- see next section) `IncludeAccounts` and `IncludeRegions` that the worker is tasked for. This outlines how we need to configure each component.
@@ -45,13 +71,86 @@ The `RecorderConfiguration` section has the following fields (if not specified a
 
 The `RecordingGroup` is defined as:
 
-1. `ResourceTypes` - This is a list of strings that will be one of 2 possible values. It will either be a list of AWS Config supported resource types **-- OR --** it will simply be a list of 1 item: `- ALL`. More on that below.
-1. `GlobalsInRegions` - This is a list of AWS regions that will record global resources. This is **only** used if `ResourceTypes` is set to a value of `- ALL`. For example, if you are recording all resource types, then you _must_ use this to define which AWS regions you want to record the global resource types (like IAM roles) in. Keep in mind, that if you record global resource types in more than 1 region, AWS will bill you duplicate times for change histories for those global resources. This field needs to be blank if you are _not_ recording `- ALL` resource types.
+1. `RecordEverything` - This is used if you want to record ALL AWS Config supported resources, including any new ones that get introduced. This is a dictionary - see more below.
+1. `RecordSpecificResources` - This is an enumerated list of AWS resources that you want to record
+1. `RecordEverythingExcept` - This is an enumerated list of AWS resources that you *don't* want to record. All other resource types are recorded including any new ones that AWS Config supports in the future.
+
+#### The `RecordingGroup` details
+The `RecordingGroup` allows you to specify which resources to record (or not record). There are 3 possible values (stated above: `RecordEverything`, `RecordSpecificResources`, or `RecordEverythingExcept`)
+that this can have. You can only define 1 of these.
+
+##### RECOMMENDED: Record all resources except for specific resources
+If you want to record all resources except for specific resources, then the schema for that looks like this:
+
+```yaml
+RecordingGroup:
+    RecordEverythingExcept:
+        - AWS::EC2::Instance
+        - AWS::EC2::Volume
+        - AWS::EC2::NetworkInterface
+```
+
+We recommend this approach since it will record all resources except for the ones you specify. This is useful to avoid recording resources that change a lot, since that can jack up your AWS Config bill.
+This allows you to record all the resources that don't excessively change a lot in your environment, which gives you the benefit of being able to track changes for the majority of your resources while keeping your bill in check.
+In the example above, we will record all resources except for EC2 Instances, EBS Volumes, and ENIs.
+
+We would also recommend having an override region where you do track the globals in.
+
+##### Record specific resources
+If you want to record specific resources, then the schema for that looks like this:
+
+```yaml
+RecordingGroup:
+    RecordSpecificResources:
+        - AWS::IAM::Role
+        - AWS::IAM::Group
+        - AWS::S3::Bucket
+```
+
+In this example, this will record only the resources specified, which in this case, are IAM Roles, IAM Groups, and S3 Buckets.
+
+##### Record ALL AWS Config supported resources
+If you want to record ALL resources, then the schema for that looks like this:
+
+```yaml
+RecordingGroup:
+    RecordEverything:
+        RecordGlobalsInTheseRegions:
+            - us-east-1
+```
+
+Under `RecordEverything` there is list called `RecordGlobalsInTheseRegions`. This is a list of regions that you want to record global resources in. There are 3 possible values for this:
+
+1. `- ALL` - If you want to record global resource types in all regions, then you can set this list to one item with the value of `- ALL`. This is not recommended since it will duplicate the recorded global resources in all the regions. This can result in a higher AWS Config bill.
+1. `- NONE` - If you do not want to record global resource types in any region, then you can set this list to one item with the value of `- NONE`. This is not recommended since you will definitely want to record things like IAM, which is global.
+1. A list of regions - If you want to record global resource types in a specific region, then you can set this list to any number of supported AWS Config regions.
+
+*Examples for all resources:*
+
+Record all resources including global resources in all regions:
+```yaml
+RecordingGroup:
+    RecordEverything:
+        RecordGlobalsInTheseRegions:
+            - ALL
+```
 
-#### `RecordingGroup` details
-The `RecordingGroup` allows you to specify which resources to record. As stated above, if you set the `ResourceTypes` field to `- ALL`, then it will set the `allResources: True` flag in the API to create or update the recorder. This will record all resources, and all newly supported resources that AWS Config adds in the future automatically. If `ResourceTypes` is set to `- ALL`, then you also need to define `GlobalsInRegions` to include the AWS regions to record the global resource types. We recommend that you set this to `us-east-1`, and optionally some other region if you would like to have multiple copies of global resources recorded.
+All resources except for global resources (i.e. don't record global resources anywhere):
+```yaml
+RecordingGroup:
+    RecordEverything:
+        RecordGlobalsInTheseRegions:
+            - NONE
+```
 
-If you don't want all resources to be recorded, then you need to set the `ResourceTypes` field to a list of AWS Config resource type names, [as defined in this page](https://docs.aws.amazon.com/config/latest/developerguide/resource-config-reference.html). Some examples are `AWS::S3::Bucket`, and `AWS::IAM::Role`.
+All resources, but only record global resources in `us-east-1` and `us-west-2`:
+```yaml
+RecordingGroup:
+    RecordEverything:
+        RecordGlobalsInTheseRegions:
+            - us-east-1
+            - us-west-2
+```
 
 ### `DeliveryChannelDetails`
 The `DeliveryChannelDetails` section defines how the delivery channel should be configured. The following fields are present (if not specified as optional, it's required):
@@ -70,13 +169,12 @@ The `DeliveryChannelDetails` section defines how the delivery channel should be
 1. `S3KmsKeyArn` - *maybe optional* - If you are using KMS with S3 to encrypt your S3 data, then you need to specify the KMS Key ARN here. This is REQUIRED if you use KMS with S3, otherwise this should not be set.
 1. `SnsTopicArn` - *optional* - This is the optional SNS topic to send notifications to whenever configuration changes are recorded. If you don't want to use this, then don't set this. See the AWS documentation for details on how this works and what it does.
 
-
 ### `RetentionConfigInDays`
 This is documented above. Basically, this is how long (in days) that AWS Config should retain configuration history. The maximum value is 2557 days, the minimum is 30. This field is required.
 
-
 ## Overriding Account Defaults
-We provide you the ability to provide Account/Region level overrides. This is useful if you need to opt-out or opt-in resource level configurations for a given set of accounts. This is set via the *OPTIONAL* `AccountOverrideConfigurations` field. The `AccountOverrideConfigurations` is a list of all the items in the `DefaultConfiguration` schema but also has the ability to specify the account and region to run and not run in.
+We provide you the ability to provide Account/Region level overrides. This is useful if you need to opt-out or opt-in resource level configurations for a given set of accounts.
+This is set via the *OPTIONAL* `AccountOverrideConfigurations` field. The `AccountOverrideConfigurations` is a list of all the items in the `DefaultConfiguration` schema but also has the ability to specify the account and region to run in.
 
 Here is an example of what a full template with this would look like:
 
@@ -96,10 +194,9 @@ DefaultConfiguration:
         ConfigRoleName: AWSConfigRole
         RecordingEnabled: True
         RecordingGroup:
-            ResourceTypes:
-                - ALL
-            GlobalsInRegions:
-                - us-east-1
+            RecordEverything:
+                RecordGlobalsInTheseRegions:
+                    - us-east-1
     RetentionPeriodInDays: 2557
 AccountOverrideConfigurations:
     -
@@ -120,7 +217,7 @@ AccountOverrideConfigurations:
             ConfigRoleName: SomeOtherRole
             RecordingEnabled: True
             RecordingGroup:
-                ResourceTypes:
+                RecordSpecificResources:
                     - AWS::S3::Bucket
                     - AWS::EC2::SecurityGroup
         RetentionPeriodInDays: 30
@@ -147,6 +244,8 @@ This section outlines some special details regarding the worker's behavior.
 1. A quick note is that when thinking about how this template is applied, you should think of it as the main template's `Include/Exclude` mostly applies to the `DefaultConfiguration`. Overrides are there to provide exceptions in specific accounts/regions under the main `Include*`. You can always create multiple templates and store them in the template S3 bucket path (default path is `AwsConfigWorker/`), however, we would recommend that you use just 1 template so everything is in one place.
 
 ## Some examples
+The most realistic example is at the top. This section shows other examples you may or may not care about.
+
 This is an example template that will enable AWS Config to all accounts and regions, including the Organization root, for all regions, for all resource types. Global resources will only be recorded in us-east-1:
 
 ```yaml
@@ -165,14 +264,14 @@ DefaultConfiguration:
         ConfigRoleName: AWSConfigRole
         RecordingEnabled: True
         RecordingGroup:
-            ResourceTypes:
-                - ALL
-            GlobalsInRegions:
-                - us-east-1
+            RecordEverything:
+                RecordGlobalsInTheseRegions:
+                    - us-east-1
     RetentionPeriodInDays: 2557
 ```
 
-This is an example that will enable AWS Config to all accounts and regions except `Account One`. In `Account Two`, only S3 buckets, and security groups will be recorded in regions not us-east-1. In `Account Two`/us-east-1, it will record S3 buckets, security groups, and IAM roles:
+This is an example that will enable AWS Config to all accounts and regions except `Account One`. In `Account Two`, only S3 buckets, and security groups will be recorded in regions not us-east-1. In `Account Two`/us-east-1, it will record S3 buckets, security groups, and IAM roles. For all other
+account/region pairs, it will record all resources except EC2 instances:
 ```yaml
 TemplateName: AWSConfigEnablement
 TemplateDescription: Enabling AWS Config
@@ -192,10 +291,8 @@ DefaultConfiguration:
         ConfigRoleName: AWSConfigRole
         RecordingEnabled: True
         RecordingGroup:
-            ResourceTypes:
-                - ALL
-            GlobalsInRegions:
-                - us-east-1
+            RecordEverythingExcept:
+                - AWS::EC2::Instance
     RetentionPeriodInDays: 2557
 AccountOverrideConfigurations:
   -
@@ -213,7 +310,7 @@ AccountOverrideConfigurations:
         ConfigRoleName: AWSConfigRole
         RecordingEnabled: True
         RecordingGroup:
-            ResourceTypes:
+            RecordSpecificResources:
                 - AWS::S3::Bucket
                 - AWS::EC2::SecurityGroup
     RetentionPeriodInDays: 2557
@@ -230,7 +327,7 @@ AccountOverrideConfigurations:
         ConfigRoleName: AWSConfigRole
         RecordingEnabled: True
         RecordingGroup:
-            ResourceTypes:
+            RecordSpecificResources:
                 - AWS::S3::Bucket
                 - AWS::EC2::SecurityGroup
                 - AWS::IAM::Role

pyproject.toml

@@ -32,7 +32,7 @@ tests = [
     "flake8==6.0.0",
     "pylint==2.17.4",
     "tox==4.5.2",
-    "moto==4.1.10",
+    "moto==4.1.12.dev59",  # Update when 4.1.12 is official
     "mkdocs==1.4.3",
     "mkdocstrings[python]==0.22.0",
     "mkdocs-gen-files==0.5.0",

src/requirements.txt

@@ -6,6 +6,6 @@ retry==0.9.2
 pyjwt==2.7.0
 requests==2.31.0
 cryptography==41.0.1
-botocore==1.29.144
-boto3==1.26.144
+botocore==1.29.157
+boto3==1.26.157
 slack-sdk==3.21.3