diff --git a/docs/admin-guide.md b/docs/admin-guide.md index a782d5e92..1f8b62c72 100644 --- a/docs/admin-guide.md +++ b/docs/admin-guide.md @@ -278,6 +278,13 @@ ensures the existence of AWS Accounts defined in `.yml` files within the or alternatively in the bootstrap repository you can find it in the `adf-accounts` directory. +**Note on Provisioning AWS Accounts via AWS Control Tower** +ADF is fully compatible with [AWS Control Tower](https://aws.amazon.com/de/controltower/). +If you deployed ADF and AWS Control Tower in your AWS Organization and if you +opted for vending AWS Accounts via AWS Control Tower, you can ignore the ADF +Account Provisioning feature. Any AWS Account vended via AWS Control Tower will +go through the regular ADF bootstrap process described below. + ### Bootstrapping Accounts #### Bootstrapping Overview diff --git a/docs/installation-guide.md b/docs/installation-guide.md index 71cc89be9..0ae5b640d 100644 --- a/docs/installation-guide.md +++ b/docs/installation-guide.md @@ -6,7 +6,28 @@ - [git](https://git-scm.com/) - [AWS CodeCommit Setup](https://docs.aws.amazon.com/codecommit/latest/userguide/setting-up-https-unixes.html) - [AWS CloudTrail configured](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-and-update-a-trail.html) -in the `us-east-1` region within the AWS Organizations Management AWS Account. + in the `us-east-1` region within the AWS Organizations Management AWS Account. + +## ADF-Compability with AWS Control Tower + +ADF is fully compatible with [AWS Control Tower](https://aws.amazon.com/de/controltower/). +ADF augments AWS Control Tower. A common operations model is defined as follows: + +- AWS Control Tower is responsible for AWS Account creation and OU mapping. +- ADF is responsible for deploying applications as defined in the ADF + deployment maps. + +In the following, we assume that you install ADF without AWS Control Tower. +However, if a specific installation step requires a "AWS Control Tower-specific +action, we call those out explicitly. + +It is okay to install ADF and AWS Control Tower in different regions. Example: + +- Install AWS Control Tower in eu-central-1. +- Install ADF in us-east-1. + +**If you want to use ADF and AWS Control Tower, we recommend that you setup +AWS Control Tower prior to installing ADF.** ## Installation Instructions @@ -18,10 +39,10 @@ in the `us-east-1` region within the AWS Organizations Management AWS Account. for AWS Organizations can only be acted upon in the US East (N. Virginia) Region. -2. In the AWS Console from your management account within `us-east-1`, head over - to the Serverless Application Repository *(SAR)*. From there, search for - `aws-deployment-framework` *(or "adf")* (ensure the checkbox *"Show apps that - create custom IAM roles or resource policies"* is checked). +2. In the AWS Console from your management account within `us-east-1`, head + over to the Serverless Application Repository *(SAR)*. From there, search + for `aws-deployment-framework` *(or "adf")* (ensure the checkbox + *"Show apps that create custom IAM roles or resource policies"* is checked). If you are deploying ADF for the first time, fill in the required parameters for your specific use-case. For example, if you have no AWS Organization @@ -30,9 +51,18 @@ in the `us-east-1` region within the AWS Organizations Management AWS Account. deployment OU, along with an AWS Account that will be used to house deployment pipelines throughout your Organization. - If you already have an AWS Account you want to use as your deployment account - you can specify its Account ID in the parameter `DeploymentAccountId` - and leave the`DeploymentAccountName` and `DeploymentAccountEmail` empty. + If you already have an AWS Account you want to use as your deployment + account you can specify its Account ID in the parameter + `DeploymentAccountId` and leave the `DeploymentAccountName` plus + `DeploymentAccountEmail` empty. + + **AWS Control Tower-specific Note:** + If you use AWS Control Tower, we recommend to create the deployment AWS + Account via the account vending feature of AWS Control Tower. + + It is **MANDATORY**, that your designated deployment AWS Account resides in + the OU `deployment` (case-sensitive!). This can't be changed currently. + Otherwise, the ADF deployment will fail! Next, specify the `DeploymentAccountMainRegion` parameter as the region that will host your deployment pipelines and would be considered your main AWS @@ -44,40 +74,61 @@ in the `us-east-1` region within the AWS Organizations Management AWS Account. Also specify a main notification endpoint *(email)* to receive updates about the bootstrap process. + **AWS Control Tower-specific Note:** + If you use AWS Control Tower, in the `CrossAccountAccessRoleName` section of + the parameters enter the string `AWSControlTowerExecution`. + Alternatively, leave empty for a default ADF setup. + When you have entered all required information press **'Deploy'**. 3. As the stack `serverlessrepo-aws-deployment-framework` completes you can now open AWS CodePipeline from within the management account in `us-east-1` and see that there is an initial pipeline execution that has been started. - When ADF is deployed for the first time, it will make the initial commit with - the skeleton structure of the `aws-deployment-framework-bootstrap` CodeCommit - repository. + When ADF is deployed for the first time, it will make the initial commit + with the skeleton structure of the `aws-deployment-framework-bootstrap` + CodeCommit repository. From that initial commit, you can clone the repository to your local environment and make the changes required to define your desired base stacks via AWS CloudFormation Templates, Service Control Policies or Tagging Policies. -4. As part of the AWS CodePipeline Execution from the previous step, the account - provisioner component will run *(in CodeBuild)*. +4. As part of the AWS CodePipeline Execution from the previous step, the + account provisioner component will run *(in CodeBuild)*. + + OPTION 4.1: ONLY applies when requesting the creation of a NEW deployment + account AND when using ADF for vending AWS Accounts. + + - If you let ADF create a new Deployment account for you + *(by not giving a pre-existing account id when deploying from SAR)*, + then ADF will handle creating and moving this account automatically into + the deployment OU. + + OPTION 4.2: ONLY applies when reusing an pre-created deployment account + AND when using ADF for vending AWS Accounts + + - If you are using a pre-existing deployment account, you will need to + move the account into the deployment OU from within the Organization + console, or add your deployment account into a `.yml` file within the + `adf-accounts` folder *(see docs)*. + + OPTION 4.3: ONLY applies when reusing a pre-existing deployment account + AND when using AWS Control Tower for vending AWS Accounts - If you let ADF create a new Deployment account for you *(by not giving a - pre-existing account id when deploying from SAR)*, then ADF will handle - creating and moving this account automatically into the deployment OU. + - Ensure that the AWS Control Tower-created deployment AWS Account + resides in the OU `deployment` (case-sensitive!). - If you are using a pre-existing deployment account, you will need to move the - account into the deployment OU from within the Organization console, or - add your deployment account into a `.yml` file within the `adf-accounts` - folder *(see docs)*. This action will trigger [AWS Step - Functions](https://aws.amazon.com/step-functions/) to run and start the - bootstrap process for the deployment account. You can view the progress of - this in the AWS Step Functions console from the management account in the + Regardless of the option taken above, after AWS Account creation, you should + see an [AWS Step Functions](https://aws.amazon.com/step-functions/) run + that started the bootstrap process for the deployment account. You can view + the progress of this in the management account in the AWS Step Functions + console for the step function `AccountBootstrappingStateMachine-` in the `us-east-1` region. 5. Once the Step Function has completed, switch roles over to the newly - bootstrapped deployment account in the region you defined as your main region - from step 2. + bootstrapped deployment account in the region you defined as your main + region from step 2. An AWS CodeCommit repository will have been created and will contain the initial skeleton structure committed which serves as a starting point for diff --git a/docs/user-guide.md b/docs/user-guide.md index 2fe192dae..1ff7e6254 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -271,10 +271,16 @@ targets: # stages, each stage containing up to X accounts size: 30 exclude: - # (Optional) List of accounts to exclude from this target. - # Currently only supports account Ids + # (Optional) List of accounts to exclude from this path. - 9999999999 properties: ... + - path: /my_ou/production/**/* # This would target all accounts in the OU path + regions: [eu-central-1, us-west-1] + name: production_step + exclude: # (Optional) List of OU Paths and Account Ids + - /my-ou/production/alpha # excludes any accounts and OUs in this path. + - 11111111111 # Excludes this account, regardless of OU path. + properties: ... ``` CodePipeline has a limit of 50 actions per stage. diff --git a/requirements-dev.txt b/requirements-dev.txt index 145b83b9a..babfea67f 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -3,5 +3,6 @@ isort==5.12.0 mock==5.1.0 pylint==2.17.4 pytest~=7.4.0 +pytest-cov==3.0.0 tox==3.28.0 yamllint==1.32.0 diff --git a/samples/sample-terraform/buildspec.yml b/samples/sample-terraform/buildspec.yml index bd458857e..81fb6aa8b 100644 --- a/samples/sample-terraform/buildspec.yml +++ b/samples/sample-terraform/buildspec.yml @@ -10,6 +10,7 @@ phases: - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet - export PATH=$PATH:$(pwd) - bash adf-build/helpers/terraform/install_terraform.sh + - pip install --upgrade pip - pip install -r adf-build/requirements.txt -q build: commands: diff --git a/src/lambda_codebase/account_processing/configure_account_regions.py b/src/lambda_codebase/account_processing/configure_account_regions.py new file mode 100644 index 000000000..94f6e3f29 --- /dev/null +++ b/src/lambda_codebase/account_processing/configure_account_regions.py @@ -0,0 +1,103 @@ +# Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: MIT-0 + +""" +Takes regions that the account is not-opted into and opts into them. +""" +from ast import literal_eval + + +import boto3 +from aws_xray_sdk.core import patch_all +from logger import configure_logger + +patch_all() +LOGGER = configure_logger(__name__) + + +def get_regions_from_ssm(ssm_client): + regions = ssm_client.get_parameter(Name="target_regions")["Parameter"].get("Value") + regions = literal_eval(regions) + return regions + + +def get_region_status(account_client, **list_region_args): + region_status_response = account_client.list_regions(**list_region_args) + region_status = { + region.get("RegionName"): region.get("RegionOptStatus") + for region in region_status_response.get("Regions") + } + # Currently no built in paginator for list_regions... + # So we have to do this manually. + next_token = region_status_response.get("NextToken") + if next_token: + while next_token: + list_region_args["NextToken"] = next_token + region_status_response = account_client.list_regions(**list_region_args) + next_token = region_status_response.get("NextToken") + region_status = region_status | { + region.get("RegionName"): region.get("RegionOptStatus") + for region in region_status_response.get("Regions") + } + return region_status + + +def enable_regions_for_account( + account_client, account_id, desired_regions, org_root_account_id +): + list_region_args = {} + enable_region_args = {} + target_is_different_account = org_root_account_id != account_id + if target_is_different_account: + list_region_args["AccountId"] = account_id + enable_region_args["AccountId"] = account_id + + region_status = get_region_status(account_client, **list_region_args) + + regions_enabled = {} + for region in desired_regions: + regions_enabled[region] = False + desired_region_status = region_status.get(region.lower()) + if not desired_region_status: + LOGGER.warning("Unable to obtain status of %s, not enabling") + if desired_region_status == "DISABLED": + LOGGER.info("Enabling Region %s because it is currently Disabled", region) + enable_region_args["RegionName"] = region.lower() + account_client.enable_region(**enable_region_args) + else: + LOGGER.info( + "Not enabling Region: %s because it is: %s", + region, + desired_region_status, + ) + if desired_region_status in ["ENABLED_BY_DEFAULT", "ENABLED"]: + regions_enabled[region] = True + LOGGER.info(regions_enabled) + return all(regions_enabled.values()) + + +def lambda_handler(event, _): + desired_regions = [] + if event.get("regions"): + LOGGER.info( + "Account Level Regions is not currently supported." + "Ignoring these values for now and using SSM only" + ) + desired_regions.extend(get_regions_from_ssm(boto3.client("ssm"))) + org_root_account_id = boto3.client("sts").get_caller_identity().get("Account") + target_account_id = event.get("account_id") + LOGGER.info( + "Target Account Id: %s - This is running in %s. These are the same: %s", + target_account_id, + org_root_account_id, + target_account_id == org_root_account_id, + ) + all_regions_enabled = enable_regions_for_account( + boto3.client("account"), + target_account_id, + desired_regions, + org_root_account_id, + ) + event["all_regions_enabled"] = all_regions_enabled + + return event diff --git a/src/lambda_codebase/account_processing/configure_account_tags.py b/src/lambda_codebase/account_processing/configure_account_tags.py index b0c3bcec4..1a2216d9f 100644 --- a/src/lambda_codebase/account_processing/configure_account_tags.py +++ b/src/lambda_codebase/account_processing/configure_account_tags.py @@ -7,37 +7,44 @@ Will not delete tags that aren't in the config file. """ - -from organizations import Organizations - import boto3 from aws_xray_sdk.core import patch_all from logger import configure_logger patch_all() LOGGER = configure_logger(__name__) +ORG_CLIENT = boto3.client("organizations") -def create_account_tags(account_id, tags, org_session: Organizations): +def create_account_tags(account_id, tags, client): LOGGER.info( "Ensuring Account: %s has tags: %s", account_id, tags, ) - org_session.create_account_tags(account_id, tags) + formatted_tags = [ + {"Key": str(key), "Value": str(value)} + for tag in tags + for key, value in tag.items() + ] + LOGGER.debug( + "Ensuring Account: %s has tags (formatted): %s", + account_id, + formatted_tags, + ) + client.tag_resource(ResourceId=account_id, Tags=formatted_tags) def lambda_handler(event, _): if event.get("tags"): - organizations = Organizations(boto3) create_account_tags( event.get("account_id"), event.get("tags"), - organizations, + ORG_CLIENT, ) else: LOGGER.info( "Account: %s does not need tags configured", - event.get('account_full_name'), + event.get("account_full_name"), ) return event diff --git a/src/lambda_codebase/account_processing/get_account_regions.py b/src/lambda_codebase/account_processing/get_account_regions.py index aac5a512f..732dda160 100644 --- a/src/lambda_codebase/account_processing/get_account_regions.py +++ b/src/lambda_codebase/account_processing/get_account_regions.py @@ -17,31 +17,33 @@ AWS_PARTITION = os.getenv("AWS_PARTITION") +def get_default_regions_for_account(ec2_client): + filtered_regions = ec2_client.describe_regions( + AllRegions=False, + Filters=[ + { + "Name": "opt-in-status", + "Values": [ + "opt-in-not-required", + "opted-in", + ], + }, + ], + )["Regions"] + default_regions = [region["RegionName"] for region in filtered_regions] + return default_regions + + def lambda_handler(event, _): - LOGGER.info("Fetching Default regions %s", event.get('account_full_name')) + LOGGER.info("Fetching Default regions %s", event.get("account_full_name")) sts = STS() account_id = event.get("account_id") role = sts.assume_cross_account_role( f"arn:{AWS_PARTITION}:iam::{account_id}:role/{ADF_ROLE_NAME}", "adf_account_get_regions", ) + default_regions = get_default_regions_for_account(role.client("ec2")) - ec2_client = role.client("ec2") - default_regions = [ - region["RegionName"] - for region in ec2_client.describe_regions( - AllRegions=False, - Filters=[ - { - "Name": "opt-in-status", - "Values": [ - "opt-in-not-required", - "opted-in" - ], - } - ], - )["Regions"] - ] LOGGER.debug("Default regions for %s: %s", account_id, default_regions) return { **event, diff --git a/src/lambda_codebase/account_processing/pytest.ini b/src/lambda_codebase/account_processing/pytest.ini index 5ee647716..77e1fbea9 100644 --- a/src/lambda_codebase/account_processing/pytest.ini +++ b/src/lambda_codebase/account_processing/pytest.ini @@ -1,2 +1,7 @@ [pytest] testpaths = tests +addopts = --cov=./src/lambda_codebase/account_processing/ --cov-fail-under=50 --cov-report term + +[coverage:run] +omit = tests/ + diff --git a/src/lambda_codebase/account_processing/tests/test_account_tags.py b/src/lambda_codebase/account_processing/tests/test_account_tags.py new file mode 100644 index 000000000..ba7b8327f --- /dev/null +++ b/src/lambda_codebase/account_processing/tests/test_account_tags.py @@ -0,0 +1,62 @@ +""" +Tests the account tag configuration lambda +""" + +import unittest +import boto3 +from botocore.stub import Stubber +from aws_xray_sdk import global_sdk_config +from ..configure_account_tags import ( + create_account_tags, +) + +global_sdk_config.set_sdk_enabled(False) + +# pylint: disable=W0106 +class SuccessTestCase(unittest.TestCase): + def test_account_tag_creation(self): + test_event = {"account_id": "123456789012", "tags": [{"CreatedBy": "ADF"}]} + ou_client = boto3.client("organizations") + stubber = Stubber(ou_client) + stubber.add_response( + "tag_resource", + {}, + { + "Tags": [{"Key": "CreatedBy", "Value": "ADF"}], + "ResourceId": "123456789012", + }, + ), + stubber.activate() + create_account_tags( + test_event.get("account_id"), test_event.get("tags"), ou_client + ) + stubber.assert_no_pending_responses() + + def test_account_tag_creation_multiple_tags(self): + test_event = { + "account_id": "123456789012", + "tags": [ + { + "CreatedBy": "ADF", + "TagName": "TagValue", + } + ], + } + ou_client = boto3.client("organizations") + stubber = Stubber(ou_client) + stubber.add_response( + "tag_resource", + {}, + { + "Tags": [ + {"Key": "CreatedBy", "Value": "ADF"}, + {"Key": "TagName", "Value": "TagValue"}, + ], + "ResourceId": "123456789012", + }, + ), + stubber.activate() + create_account_tags( + test_event.get("account_id"), test_event.get("tags"), ou_client + ) + stubber.assert_no_pending_responses() diff --git a/src/lambda_codebase/account_processing/tests/test_configure_account_regions.py b/src/lambda_codebase/account_processing/tests/test_configure_account_regions.py new file mode 100644 index 000000000..1d3b8f0ff --- /dev/null +++ b/src/lambda_codebase/account_processing/tests/test_configure_account_regions.py @@ -0,0 +1,120 @@ +""" +Tests the account alias configuration lambda +""" + +import unittest +import boto3 +from botocore.stub import Stubber +from aws_xray_sdk import global_sdk_config +from ..configure_account_regions import get_regions_from_ssm, enable_regions_for_account + +global_sdk_config.set_sdk_enabled(False) + + +class SuccessTestCase(unittest.TestCase): + def test_get_regions_from_ssm(self): + ssm_client = boto3.client("ssm", region_name="us-east-1") + ssm_stubber = Stubber(ssm_client) + ssm_stubber.add_response("get_parameter", {"Parameter": {"Value": "[1,2,3]"}}) + ssm_stubber.activate() + self.assertListEqual(get_regions_from_ssm(ssm_client), [1, 2, 3]) + + def test_enable_regions_for_account(self): + accounts_client = boto3.client("account", region_name="us-east-1") + account_stubber = Stubber(accounts_client) + account_stubber.add_response( + "list_regions", + { + "Regions": [ + {"RegionName": "us-east-1", "RegionOptStatus": "ENABLED_BY_DEFAULT"} + ] + }, + ) + account_stubber.activate() + self.assertTrue( + enable_regions_for_account( + accounts_client, + "123456789", + desired_regions=["us-east-1"], + org_root_account_id="123456789", + ) + ) + + def test_enable_regions_for_account_with_pagination(self): + accounts_client = boto3.client("account", region_name="us-east-1") + account_stubber = Stubber(accounts_client) + account_stubber.add_response( + "list_regions", + { + "Regions": [ + {"RegionName": "us-east-1", "RegionOptStatus": "ENABLED_BY_DEFAULT"} + ], + "NextToken": "1", + }, + ) + account_stubber.add_response( + "list_regions", + { + "Regions": [ + {"RegionName": "af-south-1", "RegionOptStatus": "DISABLED"} + ], + "NextToken": "2", + }, + ) + account_stubber.add_response( + "list_regions", + {"Regions": [{"RegionName": "sco-west-1", "RegionOptStatus": "DISABLED"}]}, + ) + account_stubber.add_response( + "enable_region", + {}, + {"RegionName": "af-south-1"}, + ) + account_stubber.add_response( + "enable_region", + {}, + {"RegionName": "sco-west-1"}, + ) + account_stubber.activate() + self.assertFalse( + enable_regions_for_account( + accounts_client, + "123456789", + desired_regions=["us-east-1", "af-south-1", "sco-west-1"], + org_root_account_id="123456789", + ) + ) + account_stubber.assert_no_pending_responses() + + def test_enable_regions_for_account_that_is_not_current_account(self): + accounts_client = boto3.client("account", region_name="us-east-1") + account_stubber = Stubber(accounts_client) + account_stubber.add_response( + "list_regions", + { + "Regions": [ + { + "RegionName": "us-east-1", + "RegionOptStatus": "ENABLED_BY_DEFAULT", + }, + {"RegionName": "sco-west-1", "RegionOptStatus": "DISABLED"}, + ] + }, + ) + account_stubber.add_response( + "enable_region", + {}, + { + "RegionName": "sco-west-1", + "AccountId": "123456789", + }, + ) + account_stubber.activate() + self.assertFalse( + enable_regions_for_account( + accounts_client, + "123456789", + desired_regions=["us-east-1", "sco-west-1"], + org_root_account_id="987654321", + ) + ) diff --git a/src/lambda_codebase/account_processing/tests/test_get_default_regions.py b/src/lambda_codebase/account_processing/tests/test_get_default_regions.py new file mode 100644 index 000000000..aba7cc319 --- /dev/null +++ b/src/lambda_codebase/account_processing/tests/test_get_default_regions.py @@ -0,0 +1,62 @@ +""" +Tests the account tag configuration lambda +""" + +import unittest +import boto3 +from botocore.stub import Stubber +from aws_xray_sdk import global_sdk_config +from ..get_account_regions import ( + get_default_regions_for_account, +) + +global_sdk_config.set_sdk_enabled(False) + +# pylint: disable=W0106 +class SuccessTestCase(unittest.TestCase): + @staticmethod + def test_get_default_regions_for_account(): + ec2_client = boto3.client("ec2") + stubber = Stubber(ec2_client) + stubber.add_response( + "describe_regions", + { + "Regions": [ + { + "Endpoint": "blah", + "RegionName": "us-east-1", + "OptInStatus": "opt-in-not-required", + }, + { + "Endpoint": "blah", + "RegionName": "us-east-2", + "OptInStatus": "opt-in-not-required", + }, + { + "Endpoint": "blah", + "RegionName": "us-east-3", + "OptInStatus": "opted-in", + }, + { + "Endpoint": "blah", + "RegionName": "us-east-4", + "OptInStatus": "opted-in", + }, + ] + }, + { + "AllRegions": False, + "Filters": [ + { + "Values": [ + "opt-in-not-required", + "opted-in", + ], + "Name": "opt-in-status", + }, + ], + }, + ), + stubber.activate() + regions = get_default_regions_for_account(ec2_client) + assert regions == ["us-east-1", "us-east-2", "us-east-3", "us-east-4"] diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-accounts/README.md b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-accounts/README.md index 5dfb97d87..f195d1757 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-accounts/README.md +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-accounts/README.md @@ -7,6 +7,12 @@ Organization. This process enables to end-to-end bootstrapping and associated pipeline generation for new AWS Accounts and is the recommended way to provision and setup AWS Accounts via ADF. +**Note on Provisioning AWS Accounts via AWS Control Tower** +ADF is fully compatible with [AWS Control Tower](https://aws.amazon.com/de/controltower/). +If you deployed ADF and AWS Control Tower in your AWS Organization and if you +opted for vending AWS Accounts via AWS Control Tower, you can ignore the ADF +Account Provisioning feature. + ## Overview When setting ADF for the first time you will have an auto-generated `adf.yml` diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/global.yml b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/global.yml index 6b16518bb..fb93defbb 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/global.yml +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/global.yml @@ -548,10 +548,13 @@ Resources: Statement: - Effect: Allow Sid: "AssumeRole" + Condition: + ArnEquals: + "aws:PrincipalArn": + - !GetAtt PipelineManagementApplication.Outputs.CreateRepositoryLambdaRoleArn + - !GetAtt PipelineManagementApplication.Outputs.CreateOrUpdateRuleLambdaRoleArn Principal: - AWS: - - !GetAtt PipelineManagementApplication.Outputs.CreateRepositoryLambdaRoleArn - - !GetAtt PipelineManagementApplication.Outputs.CreateOrUpdateRuleLambdaRoleArn + AWS: !Sub arn:${AWS::Partition}:iam::${AWS::AccountId}:root Action: - sts:AssumeRole Path: / diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/pipeline_management/process_deployment_map.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/pipeline_management/process_deployment_map.py index 9fcb432e4..0f538f113 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/pipeline_management/process_deployment_map.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/pipeline_management/process_deployment_map.py @@ -10,6 +10,7 @@ import json import tempfile from typing import Any, TypedDict +import hashlib import yaml from yaml.error import YAMLError @@ -173,9 +174,12 @@ def start_executions( # AWS Step Functions supports max 80 characters. # Since the run_id equals 49 characters plus the dash, we have 30 # characters available. To ensure we don't run over, lets use a - # truncated version instead: - truncated_pipeline_name = full_pipeline_name[:30] - sfn_execution_name = f"{truncated_pipeline_name}-{run_id}" + # truncated version concatenated with an hash generated from + # the pipeline name + truncated_pipeline_name = full_pipeline_name[:24] + name_bytes_to_hash = bytes(full_pipeline_name + run_id, 'utf-8') + execution_unique_hash = hashlib.md5(name_bytes_to_hash).hexdigest()[:5] + sfn_execution_name = f"{truncated_pipeline_name}-{execution_unique_hash}-{run_id}" sfn_client.start_execution( stateMachineArn=PIPELINE_MANAGEMENT_STATEMACHINE, name=sfn_execution_name, diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/example-global-iam.yml b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/example-global-iam.yml index f85d717ee..be971baec 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/example-global-iam.yml +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/example-global-iam.yml @@ -60,9 +60,13 @@ Resources: # Statement: # - Effect: Allow # Sid: "AssumeRole" +# Condition: +# ArnEquals: +# 'aws:PrincipalArn': +# # This would allow all CodeBuild projects to be able to assume this role +# - !Sub arn:${AWS::Partition}:iam::${DeploymentAccountId}:role/adf-codebuild-role # Principal: -# AWS: -# - !Sub arn:aws:iam::${DeploymentAccountId}:role/adf-codebuild-role +# AWS: !Sub arn:${AWS::Partition}:iam::${DeploymentAccountId}:root # Action: # - sts:AssumeRole # Path: / @@ -103,12 +107,16 @@ Resources: # Statement: # - Effect: Allow # Sid: "AssumeRole" +# Condition: +# ArnEquals: +# 'aws:PrincipalArn': +# # This would allow all CodeBuild projects to be able to assume this role +# # - !Sub arn:${AWS::Partition}:iam::${DeploymentAccountId}:role/adf-codebuild-role +# - !Sub arn:${AWS::Partition}:iam::${DeploymentAccountId}:role/my-custom-codebuild-role +# # The above role would be created on the deployment account +# # for the purpose deploying this custom resource via CodeBuild # Principal: -# AWS: -# # This would allow all codebuild projects to be able to assume this role -# # - !Sub arn:${AWS::Partition}:iam::${DeploymentAccountId}:role/adf-codebuild-role -# - !Sub arn:${AWS::Partition}:iam::${DeploymentAccountId}:role/my-custom-codebuild-role -# # The above role would be created on the deployment account for the purpose deploying this custom resource via codebuild +# AWS: !Sub arn:${AWS::Partition}:iam::${DeploymentAccountId}:root # Action: # - sts:AssumeRole # Path: / diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/global.yml b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/global.yml index f9aaba107..a323a94a8 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/global.yml +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/global.yml @@ -34,8 +34,11 @@ Resources: Version: "2012-10-17" Statement: - Effect: Allow + Condition: + ArnEquals: + "aws:PrincipalArn": !Sub "arn:${AWS::Partition}:iam::${DeploymentAccountId}:role/adf-codepipeline-role" Principal: - AWS: !Sub arn:${AWS::Partition}:iam::${DeploymentAccountId}:role/adf-codepipeline-role + AWS: !Sub arn:${AWS::Partition}:iam::${DeploymentAccountId}:root Action: - sts:AssumeRole - Effect: Allow @@ -154,9 +157,12 @@ Resources: Statement: - Effect: Allow Principal: - AWS: - - !Sub arn:${AWS::Partition}:iam::${DeploymentAccountId}:role/adf-codepipeline-role - - !Sub arn:${AWS::Partition}:iam::${DeploymentAccountId}:role/adf-cloudformation-role + AWS: !Sub arn:${AWS::Partition}:iam::${DeploymentAccountId}:root + Condition: + ArnEquals: + "aws:PrincipalArn": + - !Sub arn:${AWS::Partition}:iam::${DeploymentAccountId}:role/adf-codepipeline-role + - !Sub arn:${AWS::Partition}:iam::${DeploymentAccountId}:role/adf-cloudformation-role Action: - sts:AssumeRole Path: / @@ -203,12 +209,13 @@ Resources: Sid: "AssumeRole" Principal: AWS: - - !Sub arn:${AWS::Partition}:iam::${DeploymentAccountId}:role/adf-codepipeline-role + - !Sub arn:${AWS::Partition}:iam::${DeploymentAccountId}:root Action: - sts:AssumeRole Condition: ArnEquals: "aws:SourceArn": !Sub "arn:${AWS::Partition}:codepipeline:${AWS::Region}:${DeploymentAccountId}:*" + "aws:PrincipalArn": !Sub "arn:${AWS::Partition}:iam::${DeploymentAccountId}:role/adf-codepipeline-role" Path: / UpdateCrossAccountAccessByDeploymentAccountRole: @@ -220,9 +227,11 @@ Resources: Statement: - Effect: Allow Sid: "AssumeRoleByEnableCrossAccountLambda" + Condition: + ArnEquals: + "aws:PrincipalArn": !Sub arn:${AWS::Partition}:iam::${DeploymentAccountId}:role/adf-enable-cross-account-access-lambda-role Principal: - AWS: - - !Sub arn:${AWS::Partition}:iam::${DeploymentAccountId}:role/adf-enable-cross-account-access-lambda-role + AWS: !Sub arn:${AWS::Partition}:iam::${DeploymentAccountId}:root Action: - sts:AssumeRole Path: / @@ -252,10 +261,14 @@ Resources: Statement: - Effect: Allow Sid: "AssumeRole" + Condition: + ArnEquals: + "aws:PrincipalArn": + - !Sub "arn:${AWS::Partition}:iam::${DeploymentAccountId}:role/adf-automation/adf-pipeline-create-update-rule" + - !Sub "arn:${AWS::Partition}:iam::${DeploymentAccountId}:role/adf-automation/adf-pipeline-create-repository" Principal: AWS: - - !Sub arn:${AWS::Partition}:iam::${DeploymentAccountId}:role/adf-automation/adf-pipeline-create-update-rule - - !Sub arn:${AWS::Partition}:iam::${DeploymentAccountId}:role/adf-automation/adf-pipeline-create-repository + - !Sub "arn:${AWS::Partition}:iam::${DeploymentAccountId}:root" Action: - sts:AssumeRole Path: / @@ -356,9 +369,12 @@ Resources: Statement: - Effect: Allow Sid: "AssumeRole" + Condition: + ArnEquals: + "aws:PrincipalArn": !Sub "arn:${AWS::Partition}:iam::${DeploymentAccountId}:role/adf-codebuild-role" Principal: AWS: - - !Sub arn:${AWS::Partition}:iam::${DeploymentAccountId}:role/adf-codebuild-role + - !Sub arn:${AWS::Partition}:iam::${DeploymentAccountId}:root Action: - sts:AssumeRole Path: / diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/requirements-dev.txt b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/requirements-dev.txt new file mode 100644 index 000000000..5f12e99ac --- /dev/null +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/requirements-dev.txt @@ -0,0 +1 @@ +pytest-env~=0.8.2 diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/requirements.txt b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/requirements.txt index 571335443..328198a76 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/requirements.txt +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/requirements.txt @@ -1,3 +1,4 @@ boto3==1.28.8 botocore==1.31.8 docopt~=0.6.2 +schema==0.7.5 diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/organizations.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/organizations.py index 72ae8c8b9..724386ea1 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/organizations.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/organizations.py @@ -18,7 +18,11 @@ from partition import get_organization_api_region LOGGER = configure_logger(__name__) -AWS_REGION = os.getenv('AWS_REGION') +AWS_REGION = os.getenv("AWS_REGION") + + +class OrganizationsException(Exception): + pass class Organizations: # pylint: disable=R0904 @@ -32,16 +36,37 @@ class Organizations: # pylint: disable=R0904 }, ) - def __init__(self, role, account_id=None): - self.client = role.client( - 'organizations', - config=Organizations._config + def __init__( + self, role=None, account_id=None, org_client=None, tagging_client=None + ): + if role: + LOGGER.warning( + "Deprecation warning: Using a role in the organizations client is being " + "deprecated. Please provide the relevant clients to remove this warning" + ) + if not role: + if not org_client: + raise OrganizationsException( + "If a role isn't provided, please provide an org_client" + ) + if not tagging_client: + raise OrganizationsException( + "If a role isn't provided, please provide a tagging_client" + ) + self.client = ( + role.client("organizations", config=Organizations._config) + if not org_client + else org_client ) organization_api_region = get_organization_api_region(AWS_REGION) - self.tags_client = role.client( - 'resourcegroupstaggingapi', - region_name=organization_api_region, - config=Organizations._config + self.tags_client = ( + role.client( + "resourcegroupstaggingapi", + region_name=organization_api_region, + config=Organizations._config, + ) + if not tagging_client + else tagging_client ) self.account_id = account_id self.root_id = None @@ -61,13 +86,13 @@ def get_parent_info(self, account_id=None): """ response = self.list_parents(account_id or self.account_id) return { - "ou_parent_id": response.get('Id'), - "ou_parent_type": response.get('Type') + "ou_parent_id": response.get("Id"), + "ou_parent_type": response.get("Type"), } def enable_organization_policies( self, - policy_type='SERVICE_CONTROL_POLICY', + policy_type="SERVICE_CONTROL_POLICY", ): """ Enable the policies on the organization unit root id. @@ -79,22 +104,21 @@ def enable_organization_policies( """ try: self.client.enable_policy_type( - RootId=self.get_ou_root_id(), - PolicyType=policy_type + RootId=self.get_ou_root_id(), PolicyType=policy_type ) except self.client.exceptions.PolicyTypeAlreadyEnabledException: LOGGER.info( - '%s are currently enabled within the Organization', + "%s are currently enabled within the Organization", policy_type, ) @staticmethod def trim_policy_path(policy): - return policy[2:] if policy.startswith('//') else policy + return policy[2:] if policy.startswith("//") else policy @staticmethod def is_ou_id(ou_id): - return ou_id[0] in ['r', 'o'] + return ou_id[0] in ["r", "o"] def get_organization_map(self, org_structure, counter=0): for name, ou_id in org_structure.copy().items(): @@ -103,7 +127,8 @@ def get_organization_map(self, org_structure, counter=0): continue # List OUs for organization_id in [ - ou_data['Id'] for ou_data in paginator( + ou_data["Id"] + for ou_data in paginator( self.client.list_children, **{ "ParentId": ou_id, @@ -120,12 +145,13 @@ def get_organization_map(self, org_structure, counter=0): org_structure[trimmed_path] = organization_id # List accounts for account_id in [ - account_data['Id'] for account_data in paginator( + account_data["Id"] + for account_data in paginator( self.client.list_children, **{ "ParentId": ou_id, "ChildType": "ACCOUNT", - } + }, ) ]: if account_id in org_structure.values() and counter != 0: @@ -139,15 +165,13 @@ def get_organization_map(self, org_structure, counter=0): # Counter is greater than 5 here is the conditional as organizations # cannot have more than 5 levels of nested OUs + 1 accounts "level" return ( - org_structure if counter > 5 + org_structure + if counter > 5 else self.get_organization_map(org_structure, counter) ) def update_policy(self, content, policy_id): - self.client.update_policy( - PolicyId=policy_id, - Content=content - ) + self.client.update_policy(PolicyId=policy_id, Content=content) def create_policy( self, @@ -156,57 +180,51 @@ def create_policy( policy_type="SERVICE_CONTROL_POLICY", ): policy_type_name = ( - 'scp' if policy_type == "SERVICE_CONTROL_POLICY" - else 'tagging-policy' + "scp" if policy_type == "SERVICE_CONTROL_POLICY" else "tagging-policy" ) response = self.client.create_policy( Content=content, - Description=f'ADF Managed {policy_type}', - Name=f'adf-{policy_type_name}-{ou_path}', - Type=policy_type + Description=f"ADF Managed {policy_type}", + Name=f"adf-{policy_type_name}-{ou_path}", + Type=policy_type, ) - return response['Policy']['PolicySummary']['Id'] + return response["Policy"]["PolicySummary"]["Id"] @staticmethod def get_policy_body(path): - bootstrap_path = f'./adf-bootstrap/{path}' - with open(bootstrap_path, mode='r', encoding='utf-8') as policy: + bootstrap_path = f"./adf-bootstrap/{path}" + with open(bootstrap_path, mode="r", encoding="utf-8") as policy: return json.dumps(json.load(policy)) def list_policies(self, name, policy_type="SERVICE_CONTROL_POLICY"): - response = list( - paginator(self.client.list_policies, Filter=policy_type) - ) - filtered_policies = [ - policy for policy in response - if policy['Name'] == name - ] + response = list(paginator(self.client.list_policies, Filter=policy_type)) + filtered_policies = [policy for policy in response if policy["Name"] == name] if len(filtered_policies) > 0: - return filtered_policies[0]['Id'] + return filtered_policies[0]["Id"] return [] def describe_policy_id_for_target( self, target_id, - policy_type='SERVICE_CONTROL_POLICY', + policy_type="SERVICE_CONTROL_POLICY", ): response = self.client.list_policies_for_target( - TargetId=target_id, - Filter=policy_type + TargetId=target_id, Filter=policy_type ) adf_managed_policies = [ - policy for policy in response['Policies'] - if f'ADF Managed {policy_type}' in policy['Description'] + policy + for policy in response["Policies"] + if f"ADF Managed {policy_type}" in policy["Description"] ] if len(adf_managed_policies) > 0: - return adf_managed_policies[0]['Id'] + return adf_managed_policies[0]["Id"] return [] def describe_policy(self, policy_id): response = self.client.describe_policy( PolicyId=policy_id, ) - return response.get('Policy') + return response.get("Policy") def attach_policy(self, policy_id, target_id): try: @@ -218,15 +236,10 @@ def attach_policy(self, policy_id, target_id): pass def detach_policy(self, policy_id, target_id): - self.client.detach_policy( - PolicyId=policy_id, - TargetId=target_id - ) + self.client.detach_policy(PolicyId=policy_id, TargetId=target_id) def delete_policy(self, policy_id): - self.client.delete_policy( - PolicyId=policy_id - ) + self.client.delete_policy(PolicyId=policy_id) def _account_available_to_adf( self, @@ -235,9 +248,7 @@ def _account_available_to_adf( include_root, ): if protected_ou_ids or not include_root: - account_ou_id = ( - self.get_parent_info(account["Id"]).get("ou_parent_id") - ) + account_ou_id = self.get_parent_info(account["Id"]).get("ou_parent_id") if not include_root and account_ou_id.startswith("r-"): LOGGER.info( "Account %s is in the root of the AWS Organization, " @@ -292,12 +303,13 @@ def get_accounts(self, protected_ou_ids=None, include_root=True): def get_organization_info(self): response = self.client.describe_organization() return { - "organization_master_account_id": response.get( - 'Organization').get( - 'MasterAccountId' - ), - "organization_id": response.get('Organization').get('Id'), - "feature_set": response.get('Organization').get('FeatureSet') + "organization_master_account_id": ( + response + .get("Organization") + .get("MasterAccountId") + ), + "organization_id": response.get("Organization").get("Id"), + "feature_set": response.get("Organization").get("FeatureSet"), } def describe_ou_name(self, ou_id): @@ -305,7 +317,7 @@ def describe_ou_name(self, ou_id): response = self.client.describe_organizational_unit( OrganizationalUnitId=ou_id ) - return response['OrganizationalUnit']['Name'] + return response["OrganizationalUnit"]["Name"] except ClientError as error: raise RootOUIDError( "OU is the Root of the Organization", @@ -313,10 +325,8 @@ def describe_ou_name(self, ou_id): def describe_account_name(self, account_id): try: - response = self.client.describe_account( - AccountId=account_id - ) - return response['Account']['Name'] + response = self.client.describe_account(AccountId=account_id) + return response["Account"]["Name"] except ClientError as error: LOGGER.error( "Failed to retrieve account name for account ID %s", @@ -326,29 +336,23 @@ def describe_account_name(self, account_id): @staticmethod def determine_ou_path(ou_path, ou_child_name): - return f'{ou_path}/{ou_child_name}' if ou_path else ou_child_name + return f"{ou_path}/{ou_child_name}" if ou_path else ou_child_name def list_parents(self, ou_id): - return self.client.list_parents( - ChildId=ou_id - ).get('Parents')[0] + return self.client.list_parents(ChildId=ou_id).get("Parents")[0] def get_accounts_for_parent(self, parent_id): - return paginator( - self.client.list_accounts_for_parent, - ParentId=parent_id - ) + return paginator(self.client.list_accounts_for_parent, ParentId=parent_id) def get_child_ous(self, parent_id): return paginator( - self.client.list_organizational_units_for_parent, - ParentId=parent_id + self.client.list_organizational_units_for_parent, ParentId=parent_id ) def get_ou_root_id(self): - return self.client.list_roots().get('Roots')[0].get('Id') + return self.client.list_roots().get("Roots")[0].get("Id") - def dir_to_ou(self, path): + def ou_path_to_id(self, path): nested_dir_paths = path.split('/')[1:] ou_id = self.get_ou_root_id() @@ -362,8 +366,37 @@ def dir_to_ou(self, path): raise ValueError( f"Path {path} failed to return a child OU at '{nested_dir_paths[0]}'", ) - else: # pylint: disable=W0120 - return self.get_accounts_for_parent(ou_id) + return ou_id + + def dir_to_ou(self, path): + LOGGER.warning( + "Deprecation warning: The method dir_to_ou() is deprecated, " + "use get_accounts_in_path() instead" + ) + return self.get_accounts_in_path(path) + + # pylint: disable=W0102 + + def get_accounts_in_path( + self, path, resolve_children=False, ou_id=None, excluded_paths=[] + ): + ou_id = self.ou_path_to_id(path) if not ou_id else ou_id + accounts = [] + for page in self.get_accounts_for_parent(ou_id): + accounts.append(page) + if resolve_children: + LOGGER.info("Resolving OUs for %s (%s)", path, ou_id) + child_query = self.get_child_ous(ou_id) + for child in child_query: + if f"{path}/{child.get('Name')}" not in excluded_paths: + accounts.extend( + self.get_accounts_in_path( + f"{path}/{child.get('Name')}", + resolve_children, + child.get("Id"), + ) + ) + return accounts def build_account_path(self, ou_id, account_path, cache): """ @@ -372,25 +405,19 @@ def build_account_path(self, ou_id, account_path, cache): current = self.list_parents(ou_id) # While not at the root of the Organization - while current.get('Type') != "ROOT": + while current.get("Type") != "ROOT": # check cache for ou name of id - if not cache.exists(current.get('Id')): + if not cache.exists(current.get("Id")): cache.add( - current.get('Id'), - self.describe_ou_name(current.get('Id')), + current.get("Id"), + self.describe_ou_name(current.get("Id")), ) - ou_name = cache.get(current.get('Id')) + ou_name = cache.get(current.get("Id")) account_path.append(ou_name) - return self.build_account_path( - current.get('Id'), - account_path, - cache - ) + return self.build_account_path(current.get("Id"), account_path, cache) return Organizations.determine_ou_path( - '/'.join(list(reversed(account_path))), - self.describe_ou_name( - self.get_parent_info().get("ou_parent_id") - ) + "/".join(list(reversed(account_path))), + self.describe_ou_name(self.get_parent_info().get("ou_parent_id")), ) def get_account_ids_for_tags(self, tags): @@ -400,16 +427,15 @@ def get_account_ids_for_tags(self, tags): values = value else: values = [value] - tag_filter.append({'Key': key, 'Values': values}) + tag_filter.append({"Key": key, "Values": values}) account_ids = [] - paginated_resources = paginator( + for resource in paginator( self.tags_client.get_resources, TagFilters=tag_filter, - ResourceTypeFilters=['organizations'], - ) - for resource in paginated_resources: - arn = resource['ResourceARN'] - account_id = arn.split('/')[::-1][0] + ResourceTypeFilters=["organizations"], + ): + arn = resource["ResourceARN"] + account_id = arn.split("/")[::-1][0] account_ids.append(account_id) return account_ids @@ -420,14 +446,14 @@ def list_organizational_units_for_parent(self, parent_ou): self.client.get_paginator("list_organizational_units_for_parent") .paginate(ParentId=parent_ou) ) - for ou in org_units['OrganizationalUnits'] + for ou in org_units["OrganizationalUnits"] ] return organizational_units def get_account_id(self, account_name): for account in self.list_accounts(): if account["Name"].strip() == account_name.strip(): - return account['Id'] + return account["Id"] return None @@ -438,13 +464,13 @@ def list_accounts(self): existing_accounts = [ account for accounts in self.client.get_paginator("list_accounts").paginate() - for account in accounts['Accounts'] + for account in accounts["Accounts"] ] return existing_accounts def get_ou_id(self, ou_path, parent_ou_id=None): # Return root OU if '/' is provided - if ou_path.strip() == '/': + if ou_path.strip() == "/": return self.root_id # Set initial OU to start looking for given ou_path @@ -452,19 +478,19 @@ def get_ou_id(self, ou_path, parent_ou_id=None): parent_ou_id = self.root_id # Parse ou_path and find the ID - ou_hierarchy = ou_path.strip('/').split('/') + ou_hierarchy = ou_path.strip("/").split("/") hierarchy_index = 0 while hierarchy_index < len(ou_hierarchy): org_units = self.list_organizational_units_for_parent(parent_ou_id) for ou in org_units: - if ou['Name'] == ou_hierarchy[hierarchy_index]: - parent_ou_id = ou['Id'] + if ou["Name"] == ou_hierarchy[hierarchy_index]: + parent_ou_id = ou["Id"] hierarchy_index += 1 break else: raise ValueError( - f'Could not find ou with name {ou_hierarchy} in OU list {org_units}.', + f"Could not find ou with name {ou_hierarchy} in OU list {org_units}.", ) return parent_ou_id @@ -473,7 +499,7 @@ def move_account(self, account_id, ou_path): self.root_id = self.get_ou_root_id() ou_id = self.get_ou_id(ou_path) response = self.client.list_parents(ChildId=account_id) - source_parent_id = response['Parents'][0]['Id'] + source_parent_id = response["Parents"][0]["Id"] if source_parent_id == ou_id: # Account is already resided in ou_path @@ -482,23 +508,12 @@ def move_account(self, account_id, ou_path): response = self.client.move_account( AccountId=account_id, SourceParentId=source_parent_id, - DestinationParentId=ou_id - ) - - def create_account_tags(self, account_id, tags): - formatted_tags = [ - {'Key': str(key), 'Value': str(value)} - for tag in tags - for key, value in tag.items() - ] - self.client.tag_resource( - ResourceId=account_id, - Tags=formatted_tags + DestinationParentId=ou_id, ) @staticmethod def create_account_alias(account_alias, role): - iam_client = role.client('iam') + iam_client = role.client("iam") try: iam_client.create_account_alias(AccountAlias=account_alias) except iam_client.exceptions.EntityAlreadyExistsException: @@ -524,6 +539,7 @@ def create_account(self, account, adf_role_name): sleep(5) # waiting for 5 sec before checking account status again account_id = response["AccountId"] # TODO: Instead of sleeping, query for the role. - sleep(90) # Wait 90 sec until OrganizationalRole is created in new account (Temp solution) + # Wait 90 sec until OrganizationalRole is created in new account (Temp solution) + sleep(90) return account_id diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/target.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/target.py index 6bdaadfe4..a107643a3 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/target.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/target.py @@ -22,6 +22,8 @@ ADF_DEPLOYMENT_ACCOUNT_ID = os.environ["ACCOUNT_ID"] AWS_ACCOUNT_ID_REGEX = re.compile(AWS_ACCOUNT_ID_REGEX_STR) CLOUDFORMATION_PROVIDER_NAME = "cloudformation" +RECURSIVE_SUFFIX = "/**/*" + class TargetStructure: def __init__(self, target): @@ -117,7 +119,6 @@ def generate_waves(self, target): ) return waves - class Target: """ Target deployment configuration. @@ -143,25 +144,22 @@ def __init__( @staticmethod def _account_is_active(account): - return bool(account.get('Status') == 'ACTIVE') + return bool(account.get("Status") == "ACTIVE") def _create_target_info(self, name, account_id): return { "id": account_id, - "name": re.sub(r'[^A-Za-z0-9.@\-_]+', '', name), + "name": re.sub(r"[^A-Za-z0-9.@\-_]+", "", name), "path": self.path, "properties": self.properties, "provider": self.provider, "regions": self.regions, - "step_name": re.sub(r'[^A-Za-z0-9.@\-_]+', '', self.step_name) + "step_name": re.sub(r"[^A-Za-z0-9.@\-_]+", "", self.step_name), } def _target_is_approval(self): self.target_structure.account_list.append( - self._create_target_info( - 'approval', - 'approval' - ) + self._create_target_info("approval", "approval") ) def _create_response_object(self, responses): @@ -175,8 +173,7 @@ def _create_response_object(self, responses): accounts_found += 1 self.target_structure.account_list.append( self._create_target_info( - response.get('Name'), - str(response.get('Id')) + response.get("Name"), str(response.get("Id")) ) ) if accounts_found == 0: @@ -185,7 +182,7 @@ def _create_response_object(self, responses): def _target_is_account_id(self): responses = self.organizations.client.describe_account( AccountId=str(self.path) - ).get('Account') + ).get("Account") self._create_response_object([responses]) def _target_is_tags(self): @@ -205,13 +202,16 @@ def _target_is_tags(self): self._create_response_object(accounts) def _target_is_ou_id(self): - responses = self.organizations.get_accounts_for_parent( - str(self.path) - ) + responses = self.organizations.get_accounts_for_parent(str(self.path)) self._create_response_object(responses) - def _target_is_ou_path(self): - responses = self.organizations.dir_to_ou(self.path) + def _target_is_ou_path(self, resolve_children=False): + responses = self.organizations.get_accounts_in_path( + self.path, + resolve_children=resolve_children, + ou_id=None, + excluded_paths=[], + ) self._create_response_object(responses) def _target_is_null_path(self): @@ -221,11 +221,11 @@ def _target_is_null_path(self): self._create_response_object(responses) def fetch_accounts_for_target(self): - if self.path == 'approval': + if self.path == "approval": return self._target_is_approval() if isinstance(self.path, dict): return self._target_is_tags() - if str(self.path).startswith('ou-'): + if str(self.path).startswith("ou-"): return self._target_is_ou_id() if AWS_ACCOUNT_ID_REGEX.match(str(self.path)): return self._target_is_account_id() @@ -245,13 +245,13 @@ def fetch_accounts_for_target(self): # Optimistically convert the path from 10-base to octal 8-base # Then remove the use of the 'o' char, as it will output # in the correct way, starting with: 0o. - str(oct(int(self.path))).replace('o', ''), + str(oct(int(self.path))).replace("o", ""), + ) + if str(self.path).startswith("/"): + return self._target_is_ou_path( + resolve_children=str(self.path).endswith(RECURSIVE_SUFFIX) ) - if str(self.path).startswith('/'): - return self._target_is_ou_path() if self.path is None: # No path/target has been passed, path will default to /deployment return self._target_is_null_path() - raise InvalidDeploymentMapError( - f"Unknown definition for target: {self.path}" - ) + raise InvalidDeploymentMapError(f"Unknown definition for target: {self.path}") diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/tests/test_organizations.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/tests/test_organizations.py index ba47554e0..2be8ff764 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/tests/test_organizations.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/tests/test_organizations.py @@ -3,6 +3,7 @@ # pylint: skip-file +from datetime import datetime import os import boto3 @@ -10,23 +11,22 @@ from stubs import stub_organizations from mock import Mock, patch from cache import Cache -from organizations import Organizations +from organizations import Organizations, OrganizationsException +from botocore.stub import Stubber +import unittest @fixture def cls(): - return Organizations( - boto3, - '123456789012' - ) + return Organizations(boto3, "123456789012") def test_get_parent_info(cls): cls.client = Mock() cls.client.list_parents.return_value = stub_organizations.list_parents assert cls.get_parent_info() == { - "ou_parent_id": 'some_id', - "ou_parent_type": 'ORGANIZATIONAL_UNIT' + "ou_parent_id": "some_id", + "ou_parent_type": "ORGANIZATIONAL_UNIT", } cls.client.list_parents.assert_called_once_with( ChildId=cls.account_id, @@ -34,233 +34,283 @@ def test_get_parent_info(cls): def test_get_parent_info_specific_account(cls): - specific_account_id = '111111111111' + specific_account_id = "111111111111" cls.client = Mock() cls.client.list_parents.return_value = stub_organizations.list_parents assert cls.get_parent_info(specific_account_id) == { - "ou_parent_id": 'some_id', - "ou_parent_type": 'ORGANIZATIONAL_UNIT' + "ou_parent_id": "some_id", + "ou_parent_type": "ORGANIZATIONAL_UNIT", } cls.client.list_parents.assert_called_once_with( ChildId=specific_account_id, ) -@patch('organizations.paginator') +@patch("organizations.paginator") def test_get_accounts(paginator_mock, cls): all_account_ids = [ - '111111111111', - '222222222222', - '333333333333', - '444444444444', + "111111111111", + "222222222222", + "333333333333", + "444444444444", ] root_account_ids = [ - '333333333333', + "333333333333", ] cls.client = Mock() cls.client.list_parents.side_effect = lambda account_id: ( { "Id": ( - f"r-{account_id}" if account_id in root_account_ids + f"r-{account_id}" + if account_id in root_account_ids else f"ou-{account_id}" ), "Type": "ORGANIZATIONAL_UNIT", } ) - paginator_mock.return_value = list(map( - lambda account_id: ({ - "Id": account_id, - "Status": "ACTIVE", - }), - all_account_ids, - )) - assert set(map( - lambda account: account['Id'], - cls.get_accounts(), - )) == set(all_account_ids) - - -@patch('organizations.paginator') + paginator_mock.return_value = list( + map( + lambda account_id: ( + { + "Id": account_id, + "Status": "ACTIVE", + } + ), + all_account_ids, + ) + ) + assert set( + map( + lambda account: account["Id"], + cls.get_accounts(), + ) + ) == set(all_account_ids) + + +@patch("organizations.paginator") def test_get_accounts_with_suspended(paginator_mock, cls): all_account_ids = [ - '111111111111', - '222222222222', - '333333333333', - '444444444444', + "111111111111", + "222222222222", + "333333333333", + "444444444444", ] root_account_ids = [ - '333333333333', + "333333333333", ] suspended_account_ids = [ - '444444444444', + "444444444444", ] cls.client = Mock() cls.client.list_parents.side_effect = lambda account_id: ( { "Id": ( - f"r-{account_id}" if account_id in root_account_ids + f"r-{account_id}" + if account_id in root_account_ids else f"ou-{account_id}" ), "Type": "ORGANIZATIONAL_UNIT", } ) - paginator_mock.return_value = list(map( - lambda account_id: ({ - "Id": account_id, - "Status": ( - "SUSPENDED" if account_id in suspended_account_ids - else "ACTIVE" + paginator_mock.return_value = list( + map( + lambda account_id: ( + { + "Id": account_id, + "Status": ( + "SUSPENDED" if account_id in suspended_account_ids else "ACTIVE" + ), + } ), - }), - all_account_ids, - )) - assert set(map( - lambda account: account['Id'], - cls.get_accounts(), - )) == (set(all_account_ids) - set(suspended_account_ids)) + all_account_ids, + ) + ) + assert set( + map( + lambda account: account["Id"], + cls.get_accounts(), + ) + ) == (set(all_account_ids) - set(suspended_account_ids)) -@patch('organizations.paginator') +@patch("organizations.paginator") def test_get_accounts_ignore_root(paginator_mock, cls): all_account_ids = [ - '111111111111', - '222222222222', - '333333333333', - '444444444444', + "111111111111", + "222222222222", + "333333333333", + "444444444444", ] root_account_ids = [ - '444444444444', + "444444444444", ] cls.client = Mock() - cls.client.list_parents.side_effect = lambda ChildId: ({ - "Parents": [{ - "Id": ( - f"r-{ChildId}" if ChildId in root_account_ids - else f"ou-{ChildId}" + cls.client.list_parents.side_effect = lambda ChildId: ( + { + "Parents": [ + { + "Id": ( + f"r-{ChildId}" + if ChildId in root_account_ids + else f"ou-{ChildId}" + ), + "Type": "ORGANIZATIONAL_UNIT", + } + ], + } + ) + paginator_mock.return_value = list( + map( + lambda account_id: ( + { + "Id": account_id, + "Status": "ACTIVE", + } ), - "Type": "ORGANIZATIONAL_UNIT", - }], - }) - paginator_mock.return_value = list(map( - lambda account_id: ({ - "Id": account_id, - "Status": "ACTIVE", - }), - all_account_ids, - )) - assert set(map( - lambda account: account['Id'], - cls.get_accounts( - include_root=False, - ), - )) == (set(all_account_ids) - set(root_account_ids)) - - -@patch('organizations.paginator') + all_account_ids, + ) + ) + assert set( + map( + lambda account: account["Id"], + cls.get_accounts( + include_root=False, + ), + ) + ) == (set(all_account_ids) - set(root_account_ids)) + + +@patch("organizations.paginator") def test_get_accounts_ignore_protected(paginator_mock, cls): all_account_ids = [ - '111111111111', - '222222222222', - '333333333333', - '444444444444', + "111111111111", + "222222222222", + "333333333333", + "444444444444", ] root_account_ids = [ - '444444444444', + "444444444444", ] protected_account_ids = [ - '222222222222', + "222222222222", ] - protected_ou_ids = list(map( - lambda account_id: f"ou-{account_id}", - protected_account_ids, - )) + protected_ou_ids = list( + map( + lambda account_id: f"ou-{account_id}", + protected_account_ids, + ) + ) cls.client = Mock() - cls.client.list_parents.side_effect = lambda ChildId: ({ - "Parents": [{ - "Id": ( - f"r-{ChildId}" if ChildId in root_account_ids - else f"ou-{ChildId}" + cls.client.list_parents.side_effect = lambda ChildId: ( + { + "Parents": [ + { + "Id": ( + f"r-{ChildId}" + if ChildId in root_account_ids + else f"ou-{ChildId}" + ), + "Type": "ORGANIZATIONAL_UNIT", + } + ], + } + ) + paginator_mock.return_value = list( + map( + lambda account_id: ( + { + "Id": account_id, + "Status": "ACTIVE", + } ), - "Type": "ORGANIZATIONAL_UNIT", - }], - }) - paginator_mock.return_value = list(map( - lambda account_id: ({ - "Id": account_id, - "Status": "ACTIVE", - }), - all_account_ids, - )) - assert set(map( - lambda account: account['Id'], - cls.get_accounts( - protected_ou_ids=protected_ou_ids, - ), - )) == (set(all_account_ids) - set(protected_account_ids)) - - -@patch('organizations.paginator') + all_account_ids, + ) + ) + assert set( + map( + lambda account: account["Id"], + cls.get_accounts( + protected_ou_ids=protected_ou_ids, + ), + ) + ) == (set(all_account_ids) - set(protected_account_ids)) + + +@patch("organizations.paginator") def test_get_accounts_ignore_root_protected_and_inactive(paginator_mock, cls): all_account_ids = [ - '111111111111', - '222222222222', - '333333333333', - '444444444444', - '555555555555', - '666666666666', - '777777777777', - '888888888888', + "111111111111", + "222222222222", + "333333333333", + "444444444444", + "555555555555", + "666666666666", + "777777777777", + "888888888888", ] protected_account_ids = [ - '222222222222', - '777777777777', + "222222222222", + "777777777777", ] root_account_ids = [ - '333333333333', - '888888888888', + "333333333333", + "888888888888", ] suspended_account_ids = [ - '444444444444', + "444444444444", ] pending_closure_account_ids = [ - '555555555555', + "555555555555", ] - protected_ou_ids = list(map( - lambda account_id: f"ou-{account_id}", - protected_account_ids, - )) + protected_ou_ids = list( + map( + lambda account_id: f"ou-{account_id}", + protected_account_ids, + ) + ) cls.client = Mock() - cls.client.list_parents.side_effect = lambda ChildId: ({ - "Parents": [{ - "Id": ( - f"r-{ChildId}" if ChildId in root_account_ids - else f"ou-{ChildId}" + cls.client.list_parents.side_effect = lambda ChildId: ( + { + "Parents": [ + { + "Id": ( + f"r-{ChildId}" + if ChildId in root_account_ids + else f"ou-{ChildId}" + ), + "Type": "ORGANIZATIONAL_UNIT", + } + ], + } + ) + paginator_mock.return_value = list( + map( + lambda account_id: ( + { + "Id": account_id, + "Status": ( + "SUSPENDED" + if account_id in suspended_account_ids + else ( + "PENDING_CLOSURE" + if account_id in pending_closure_account_ids + else "ACTIVE" + ) + ), + } ), - "Type": "ORGANIZATIONAL_UNIT", - }], - }) - paginator_mock.return_value = list(map( - lambda account_id: ({ - "Id": account_id, - "Status": ( - "SUSPENDED" - if account_id in suspended_account_ids - else ( - "PENDING_CLOSURE" - if account_id in pending_closure_account_ids - else "ACTIVE" - ) + all_account_ids, + ) + ) + assert set( + map( + lambda account: account["Id"], + cls.get_accounts( + protected_ou_ids=protected_ou_ids, + include_root=False, ), - }), - all_account_ids, - )) - assert set(map( - lambda account: account['Id'], - cls.get_accounts( - protected_ou_ids=protected_ou_ids, - include_root=False, - ), - )) == ( + ) + ) == ( set(all_account_ids) - set(protected_account_ids) - set(root_account_ids) @@ -275,9 +325,9 @@ def test_get_organization_info(cls): stub_organizations.describe_organization ) assert cls.get_organization_info() == { - 'organization_id': 'some_org_id', - 'organization_master_account_id': 'some_master_account_id', - 'feature_set': 'ALL' + "organization_id": "some_org_id", + "organization_master_account_id": "some_master_account_id", + "feature_set": "ALL", } @@ -286,25 +336,23 @@ def test_describe_ou_name(cls): cls.client.describe_organizational_unit.return_value = ( stub_organizations.describe_organizational_unit ) - assert cls.describe_ou_name('some_ou_id') == 'some_ou_name' + assert cls.describe_ou_name("some_ou_id") == "some_ou_name" def test_describe_account_name(cls): cls.client = Mock() - cls.client.describe_account.return_value = ( - stub_organizations.describe_account - ) - assert cls.describe_account_name('some_account_id') == 'some_account_name' + cls.client.describe_account.return_value = stub_organizations.describe_account + assert cls.describe_account_name("some_account_id") == "some_account_name" def test_determine_ou_path(cls): - assert cls.determine_ou_path( - 'some_path', 'some_ou_name' - ) == 'some_path/some_ou_name' - assert cls.determine_ou_path( - 'some_path/longer_path/plus_more', - 'some_ou_name' - ) == 'some_path/longer_path/plus_more/some_ou_name' + assert ( + cls.determine_ou_path("some_path", "some_ou_name") == "some_path/some_ou_name" + ) + assert ( + cls.determine_ou_path("some_path/longer_path/plus_more", "some_ou_name") + == "some_path/longer_path/plus_more/some_ou_name" + ) def test_build_account_path(cls): @@ -314,5 +362,491 @@ def test_build_account_path(cls): cls.client.describe_organizational_unit.return_value = ( stub_organizations.describe_organizational_unit ) + assert cls.build_account_path("some_ou_id", [], cache) == "some_ou_name" + + +class OUPathsHappyTestCases(unittest.TestCase): + """ + These test cases all use the same org structure + /production + No Accounts + /production/banking + Accounts: 111111111, 22222222 + /production/banking/investment + Accounts: 333333333 + + """ + + def test_original_ou_paths(self): + org_client = boto3.client("organizations") + org_stubber = Stubber(org_client) + tagging_client = boto3.client("organizations") + tag_stubber = Stubber(tagging_client) + + list_roots_response = { + "Roots": [ + { + "Id": "r-1337", + "Arn": "arn:aws:organizations::root/r-1337", + "Name": "/", + "PolicyTypes": [], + } + ] + } + + list_organizational_units_for_root_response = { + "OrganizationalUnits": [ + {"Id": "ou-123456", "Arn": "", "Name": "production"} + ] + } + + list_organizational_units_for_production_response = { + "OrganizationalUnits": [{"Id": "ou-080922", "Arn": "", "Name": "banking"}] + } + + list_organizational_units_for_banking_response = { + "OrganizationalUnits": [ + {"Id": "ou-09092022", "Arn": "", "Name": "investment"} + ] + } + + list_organizational_units_for_investment_response = {"OrganizationalUnits": []} + + list_accounts_for_banking_response_page_0 = { + "Accounts": [ + { + "Id": "11111111111", + "Arn": "", + "Email": "account+1@example.com", + "Status": "ACTIVE", + "JoinedMethod": "Invited", + "JoinedTimestamp": datetime(2022, 8, 9), + } + ], + "NextToken": "PAGE1", + } + list_accounts_for_banking_response_page_1 = { + "Accounts": [ + { + "Id": "22222222222", + "Arn": "", + "Email": "account+2@example.com", + "Status": "ACTIVE", + "JoinedMethod": "Invited", + "JoinedTimestamp": datetime(2022, 8, 9), + } + ] + } + + list_accounts_for_investment_response_page_0 = { + "Accounts": [ + { + "Id": "3333333333", + "Arn": "", + "Email": "account+3@example.com", + "Status": "ACTIVE", + "JoinedMethod": "Invited", + "JoinedTimestamp": datetime(2022, 8, 9), + } + ] + } + + expected_response = [ + { + "Id": "11111111111", + "Arn": "", + "Email": "account+1@example.com", + "Status": "ACTIVE", + "JoinedMethod": "Invited", + "JoinedTimestamp": datetime(2022, 8, 9), + }, + { + "Id": "22222222222", + "Arn": "", + "Email": "account+2@example.com", + "Status": "ACTIVE", + "JoinedMethod": "Invited", + "JoinedTimestamp": datetime(2022, 8, 9), + }, + ] + + org_stubber.add_response("list_roots", list_roots_response) + org_stubber.add_response( + "list_organizational_units_for_parent", + list_organizational_units_for_root_response, + {"ParentId": "r-1337"}, + ) + org_stubber.add_response( + "list_organizational_units_for_parent", + list_organizational_units_for_production_response, + {"ParentId": "ou-123456"}, + ) + org_stubber.add_response( + "list_accounts_for_parent", + list_accounts_for_banking_response_page_0, + {"ParentId": "ou-080922"}, + ) + org_stubber.add_response( + "list_accounts_for_parent", + list_accounts_for_banking_response_page_1, + {"ParentId": "ou-080922", "NextToken": "PAGE1"}, + ) + org_stubber.add_response( + "list_organizational_units_for_parent", + list_organizational_units_for_banking_response, + {"ParentId": "ou-080922"}, + ) + org_stubber.add_response( + "list_accounts_for_parent", + list_accounts_for_investment_response_page_0, + {"ParentId": "ou-09092022"}, + ) + org_stubber.add_response( + "list_organizational_units_for_parent", + list_organizational_units_for_investment_response, + {"ParentId": "ou-09092022"}, + ) + org_stubber.add_response( + "list_accounts_for_parent", + list_accounts_for_investment_response_page_0, + {"ParentId": "ou-09092022"}, + ) + + org_stubber.activate() + tag_stubber.activate() + organizations = Organizations( + role=None, org_client=org_client, tagging_client=tagging_client + ) + response = organizations.dir_to_ou("/production/banking") + + self.assertListEqual(expected_response, list(response)) + + def test_original_nested_paths(self): + org_client = boto3.client("organizations") + org_stubber = Stubber(org_client) + tagging_client = boto3.client("organizations") + tag_stubber = Stubber(tagging_client) + + list_roots_response = { + "Roots": [ + { + "Id": "r-1337", + "Arn": "arn:aws:organizations::root/r-1337", + "Name": "/", + "PolicyTypes": [], + } + ] + } + + list_organizational_units_for_root_response = { + "OrganizationalUnits": [ + {"Id": "ou-123456", "Arn": "", "Name": "production"} + ] + } + + list_organizational_units_for_production_response = { + "OrganizationalUnits": [{"Id": "ou-080922", "Arn": "", "Name": "banking"}] + } + + list_organizational_units_for_banking_response = { + "OrganizationalUnits": [ + {"Id": "ou-09092022", "Arn": "", "Name": "investment"} + ] + } + + list_organizational_units_for_investment_response = {"OrganizationalUnits": []} + + list_accounts_for_banking_response_page_0 = { + "Accounts": [ + { + "Id": "11111111111", + "Arn": "", + "Email": "account+1@example.com", + "Status": "ACTIVE", + "JoinedMethod": "Invited", + "JoinedTimestamp": datetime(2022, 8, 9), + } + ], + "NextToken": "PAGE1", + } + list_accounts_for_banking_response_page_1 = { + "Accounts": [ + { + "Id": "22222222222", + "Arn": "", + "Email": "account+2@example.com", + "Status": "ACTIVE", + "JoinedMethod": "Invited", + "JoinedTimestamp": datetime(2022, 8, 9), + } + ] + } + + list_accounts_for_investment_response_page_0 = { + "Accounts": [ + { + "Id": "3333333333", + "Arn": "", + "Email": "account+3@example.com", + "Status": "ACTIVE", + "JoinedMethod": "Invited", + "JoinedTimestamp": datetime(2022, 8, 9), + } + ] + } + + expected_response = [ + { + "Id": "11111111111", + "Arn": "", + "Email": "account+1@example.com", + "Status": "ACTIVE", + "JoinedMethod": "Invited", + "JoinedTimestamp": datetime(2022, 8, 9), + }, + { + "Id": "22222222222", + "Arn": "", + "Email": "account+2@example.com", + "Status": "ACTIVE", + "JoinedMethod": "Invited", + "JoinedTimestamp": datetime(2022, 8, 9), + }, + { + "Id": "3333333333", + "Arn": "", + "Email": "account+3@example.com", + "Status": "ACTIVE", + "JoinedMethod": "Invited", + "JoinedTimestamp": datetime(2022, 8, 9), + }, + ] + + org_stubber.add_response("list_roots", list_roots_response) + org_stubber.add_response( + "list_organizational_units_for_parent", + list_organizational_units_for_root_response, + {"ParentId": "r-1337"}, + ) + org_stubber.add_response( + "list_organizational_units_for_parent", + list_organizational_units_for_production_response, + {"ParentId": "ou-123456"}, + ) + org_stubber.add_response( + "list_accounts_for_parent", + list_accounts_for_banking_response_page_0, + {"ParentId": "ou-080922"}, + ) + org_stubber.add_response( + "list_accounts_for_parent", + list_accounts_for_banking_response_page_1, + {"ParentId": "ou-080922", "NextToken": "PAGE1"}, + ) + org_stubber.add_response( + "list_organizational_units_for_parent", + list_organizational_units_for_banking_response, + {"ParentId": "ou-080922"}, + ) + org_stubber.add_response( + "list_accounts_for_parent", + list_accounts_for_investment_response_page_0, + {"ParentId": "ou-09092022"}, + ) + org_stubber.add_response( + "list_organizational_units_for_parent", + list_organizational_units_for_investment_response, + {"ParentId": "ou-09092022"}, + ) + org_stubber.add_response( + "list_accounts_for_parent", + list_accounts_for_investment_response_page_0, + {"ParentId": "ou-09092022"}, + ) + + org_stubber.activate() + tag_stubber.activate() + organizations = Organizations( + role=None, org_client=org_client, tagging_client=tagging_client + ) + response = organizations.get_accounts_in_path( + "/production/banking", resolve_children=True + ) + + self.assertListEqual(expected_response, list(response)) + + def test_nested_paths_with_exclusions(self): + org_client = boto3.client("organizations") + org_stubber = Stubber(org_client) + tagging_client = boto3.client("organizations") + tag_stubber = Stubber(tagging_client) + + list_roots_response = { + "Roots": [ + { + "Id": "r-1337", + "Arn": "arn:aws:organizations::root/r-1337", + "Name": "/", + "PolicyTypes": [], + } + ] + } + + list_organizational_units_for_root_response = { + "OrganizationalUnits": [ + {"Id": "ou-123456", "Arn": "", "Name": "production"} + ] + } + + list_organizational_units_for_production_response = { + "OrganizationalUnits": [{"Id": "ou-080922", "Arn": "", "Name": "banking"}] + } + + list_organizational_units_for_banking_response = { + "OrganizationalUnits": [ + {"Id": "ou-09092022", "Arn": "", "Name": "investment"}, + {"Id": "ou-26092022", "Arn": "", "Name": "commercial"}, + ] + } + + list_organizational_units_for_investment_response = {"OrganizationalUnits": []} + + list_organizational_units_for_commercial_response = {"OrganizationalUnits": []} + + list_accounts_for_banking_response_page_0 = { + "Accounts": [ + { + "Id": "11111111111", + "Arn": "", + "Email": "account+1@example.com", + "Status": "ACTIVE", + "JoinedMethod": "Invited", + "JoinedTimestamp": datetime(2022, 8, 9), + } + ], + "NextToken": "PAGE1", + } + list_accounts_for_banking_response_page_1 = { + "Accounts": [ + { + "Id": "22222222222", + "Arn": "", + "Email": "account+2@example.com", + "Status": "ACTIVE", + "JoinedMethod": "Invited", + "JoinedTimestamp": datetime(2022, 8, 9), + } + ] + } + + list_accounts_for_investment_response_page_0 = { + "Accounts": [ + { + "Id": "3333333333", + "Arn": "", + "Email": "account+3@example.com", + "Status": "ACTIVE", + "JoinedMethod": "Invited", + "JoinedTimestamp": datetime(2022, 8, 9), + } + ] + } + + list_accounts_for_commercial_response_page_0 = { + "Accounts": [ + { + "Id": "444444444", + "Arn": "", + "Email": "account+4@example.com", + "Status": "ACTIVE", + "JoinedMethod": "Invited", + "JoinedTimestamp": datetime(2022, 9, 26), + } + ] + } + + expected_response = [ + { + "Id": "11111111111", + "Arn": "", + "Email": "account+1@example.com", + "Status": "ACTIVE", + "JoinedMethod": "Invited", + "JoinedTimestamp": datetime(2022, 8, 9), + }, + { + "Id": "22222222222", + "Arn": "", + "Email": "account+2@example.com", + "Status": "ACTIVE", + "JoinedMethod": "Invited", + "JoinedTimestamp": datetime(2022, 8, 9), + }, + { + "Id": "444444444", + "Arn": "", + "Email": "account+4@example.com", + "Status": "ACTIVE", + "JoinedMethod": "Invited", + "JoinedTimestamp": datetime(2022, 9, 26), + }, + ] + + org_stubber.add_response("list_roots", list_roots_response) + org_stubber.add_response( + "list_organizational_units_for_parent", + list_organizational_units_for_root_response, + {"ParentId": "r-1337"}, + ) + org_stubber.add_response( + "list_organizational_units_for_parent", + list_organizational_units_for_production_response, + {"ParentId": "ou-123456"}, + ) + org_stubber.add_response( + "list_accounts_for_parent", + list_accounts_for_banking_response_page_0, + {"ParentId": "ou-080922"}, + ) + org_stubber.add_response( + "list_accounts_for_parent", + list_accounts_for_banking_response_page_1, + {"ParentId": "ou-080922", "NextToken": "PAGE1"}, + ) + org_stubber.add_response( + "list_organizational_units_for_parent", + list_organizational_units_for_banking_response, + {"ParentId": "ou-080922"}, + ) + + org_stubber.add_response( + "list_accounts_for_parent", + list_accounts_for_commercial_response_page_0, + {"ParentId": "ou-26092022"}, + ) + + org_stubber.add_response( + "list_organizational_units_for_parent", + list_organizational_units_for_commercial_response, + {"ParentId": "ou-26092022"}, + ) + + org_stubber.activate() + tag_stubber.activate() + organizations = Organizations( + role=None, org_client=org_client, tagging_client=tagging_client + ) + response = organizations.get_accounts_in_path( + "/production/banking", + resolve_children=True, + excluded_paths=["/production/banking/investment"], + ) + + self.assertListEqual(expected_response, list(response)) + - assert cls.build_account_path('some_ou_id', [], cache) == 'some_ou_name' +class OrgClientInitTestCases(unittest.TestCase): + def test_init(self): + with self.assertRaises(OrganizationsException) as context: + Organizations() + assert Organizations(role=boto3) diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/tests/test_schema_validation.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/tests/test_schema_validation.py new file mode 100644 index 000000000..683122b83 --- /dev/null +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/tests/test_schema_validation.py @@ -0,0 +1,230 @@ +""" +Tests for schema validation +""" + +import unittest +import schema_validation +from schema import Schema + + +class NotificationSchemaValidationHappyPaths(unittest.TestCase): + def test_notification_props_schema_lambda(self): + notification_props = {"target": "my_cool_target", "type": "lambda"} + self.assertDictEqual( + Schema(schema_validation.NOTIFICATION_PROPS).validate(notification_props), + notification_props, + ) + + def test_notification_props_schema_chatbot(self): + notification_props = {"target": "my_cool_target", "type": "chat_bot"} + self.assertDictEqual( + Schema(schema_validation.NOTIFICATION_PROPS).validate(notification_props), + notification_props, + ) + + def test_param_schema_default(self): + param_props = { + "notification_endpoint": "a_notification_endpoint", + "schedule": "a_schedule_string", + "restart_execution_on_update": True, + } + + expected_response = {**param_props, "pipeline_type": "default"} + + self.assertDictEqual( + Schema(schema_validation.PARAM_SCHEMA).validate(param_props), + expected_response, + ) + + def test_param_schema_pipeline_type(self): + param_props = { + "notification_endpoint": "a_notification_endpoint", + "schedule": "a_schedule_string", + "restart_execution_on_update": True, + "pipeline_type": "default", + } + self.assertDictEqual( + Schema(schema_validation.PARAM_SCHEMA).validate(param_props), + param_props, + ) + + def test_param_schema_pipeline_type_with_notification_props(self): + param_props = { + "notification_endpoint": { + "target": "#slackchannel", + "type": "chat_bot", + }, + "schedule": "a_schedule_string", + "restart_execution_on_update": True, + "pipeline_type": "default", + } + self.assertDictEqual( + Schema(schema_validation.PARAM_SCHEMA).validate(param_props), + param_props, + ) + + +class CodeCommitSchemaValidationHappyPaths(unittest.TestCase): + def test_codecommit_source_props_schema_default(self): + codecommit_props = { + "account_id": "111111111111", + "repository": "a_repo_name", + "branch": "mainline", + "poll_for_changes": True, + "owner": "a_repo_owner", + "role": "a_role_name", + "trigger_on_changes": True, + } + expected_result = {**codecommit_props, "output_artifact_format": None} + self.assertDictEqual( + Schema(schema_validation.CODECOMMIT_SOURCE_PROPS).validate( + codecommit_props + ), + expected_result, + ) + + def test_codecommit_source_props_schema_output_format_clone_ref(self): + codecommit_props = { + "account_id": "111111111111", + "repository": "a_repo_name", + "branch": "mainline", + "poll_for_changes": True, + "owner": "a_repo_owner", + "role": "a_role_name", + "trigger_on_changes": True, + "output_artifact_format": "CODEBUILD_CLONE_REF", + } + self.assertDictEqual( + Schema(schema_validation.CODECOMMIT_SOURCE_PROPS).validate( + codecommit_props + ), + codecommit_props, + ) + + def test_codecommit_source_props_schema_output_format_code_zip(self): + codecommit_props = { + "account_id": "111111111111", + "repository": "a_repo_name", + "branch": "mainline", + "poll_for_changes": True, + "owner": "a_repo_owner", + "role": "a_role_name", + "trigger_on_changes": True, + "output_artifact_format": "CODE_ZIP", + } + self.assertDictEqual( + Schema(schema_validation.CODECOMMIT_SOURCE_PROPS).validate( + codecommit_props + ), + codecommit_props, + ) + + def test_codecommit_source_schema(self): + codecommit_props = { + "account_id": "111111111111", + "repository": "a_repo_name", + "branch": "mainline", + "poll_for_changes": True, + "owner": "a_repo_owner", + "role": "a_role_name", + "trigger_on_changes": True, + "output_artifact_format": "CODE_ZIP", + } + codecommit_source = {"provider": "codecommit", "properties": codecommit_props} + self.assertDictEqual( + Schema(schema_validation.CODECOMMIT_SOURCE).validate(codecommit_source), + codecommit_source, + ) + + +class GithubSchemaValidationHappyPaths(unittest.TestCase): + def test_github_source_props_schema_default(self): + source_props = { + "repository": "a_repo_name", + "branch": "mainline", + "owner": "a_repo_owner", + "oauth_token_path": "a_token_path", + "json_field": "a_json_field", + "trigger_on_changes": True, + } + self.assertDictEqual( + Schema(schema_validation.GITHUB_SOURCE_PROPS).validate(source_props), + source_props, + ) + + def test_github_source_schema_default(self): + source_props = { + "repository": "a_repo_name", + "branch": "mainline", + "owner": "a_repo_owner", + "oauth_token_path": "a_token_path", + "json_field": "a_json_field", + "trigger_on_changes": True, + } + + github_source = {"provider": "github", "properties": source_props} + + self.assertDictEqual( + Schema(schema_validation.GITHUB_SOURCE).validate(github_source), + github_source, + ) + + +class TargetSchemaValidationHappyPaths(unittest.TestCase): + def test_target_list_schema_list(self): + target_value = ["11111111111", "2222222222"] + self.assertEqual( + Schema(schema_validation.TARGET_LIST_SCHEMA).validate(target_value), + target_value, + ) + + def test_target_list_wave_schema_default(self): + wave_schema = {"size": 50} + self.assertDictEqual( + Schema(schema_validation.TARGET_WAVE_SCHEME).validate({}), + wave_schema, + ) + + def test_target_list_wave_schema_with_value(self): + wave_schema = {"size": 30} + self.assertDictEqual( + Schema(schema_validation.TARGET_WAVE_SCHEME).validate(wave_schema), + wave_schema, + ) + + def test_target_schema_defaults(self): + target_schema = {"wave": {"size": 50}, "exclude": []} + self.assertDictEqual( + Schema(schema_validation.TARGET_SCHEMA).validate({}), + target_schema, + ) + + def test_target_schema_configured_path(self): + target_schema = { + "path": [ + "/some_org/production/banking", + ], + "wave": { + "size": 50, + }, + "exclude": [], + } + self.assertDictEqual( + Schema(schema_validation.TARGET_SCHEMA).validate(target_schema), + target_schema, + ) + + def test_target_schema_configured_path_for_recursive_ous(self): + target_schema = { + "path": [ + "/some_org/production/*", + ], + "wave": { + "size": 50, + }, + "exclude": ["/some_org/production/"], + } + self.assertDictEqual( + Schema(schema_validation.TARGET_SCHEMA).validate(target_schema), + target_schema, + ) diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/tests/test_target.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/tests/test_target.py index 704ce523c..694bb7e75 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/tests/test_target.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/tests/test_target.py @@ -21,7 +21,9 @@ class MockOrgClient: def __init__(self, return_value) -> None: self.values = return_value - def dir_to_ou(self, path): + def get_accounts_in_path( + self, path, ou_id, resolve_children=False, excluded_paths=[] + ): return self.values @@ -55,7 +57,7 @@ def test_fetch_accounts_for_target_ou_path(): with patch.object(cls, "_target_is_ou_path") as mock: cls.fetch_accounts_for_target() - mock.assert_called_once_with() + mock.assert_called_once_with(resolve_children=False) def test_fetch_accounts_for_target_account_id(): diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/pytest.ini b/src/lambda_codebase/initial_commit/bootstrap_repository/pytest.ini index d31fa8349..bafa7f6ef 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/pytest.ini +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/pytest.ini @@ -1,3 +1,5 @@ [pytest] -testpaths = adf-build/tests adf-bootstrap/deployment/lambda_codebase/tests +env = + ACCOUNT_ID="123456789012" +testpaths = adf-build/tests adf-bootstrap/deployment/lambda_codebase/tests adf-build/shared/python/tests/ norecursedirs = adf-bootstrap/deployment/lambda_codebase/initial_commit adf-bootstrap/deployment/lambda_codebase/determine_default_branch adf-build/shared diff --git a/src/template.yml b/src/template.yml index c1658f207..bf6430d51 100644 --- a/src/template.yml +++ b/src/template.yml @@ -263,6 +263,7 @@ Resources: - !Ref GetAccountRegionsFunctionRole - !Ref DeleteDefaultVPCFunctionRole - !Ref AccountAliasConfigFunctionRole + - !Ref AccountRegionConfigFunctionRole - !Ref AccountTagConfigFunctionRole - !Ref AccountOUConfigFunctionRole - !Ref CreateAccountFunctionRole @@ -301,6 +302,7 @@ Resources: - !GetAtt AccountOUConfigFunction.Arn - !GetAtt GetAccountRegionsFunction.Arn - !GetAtt DeleteDefaultVPCFunction.Arn + - !GetAtt AccountRegionConfigFunction.Arn AccountFileProcessingFunction: Type: 'AWS::Serverless::Function' @@ -425,6 +427,54 @@ Resources: FunctionName: AccountTagConfigurationFunction Role: !GetAtt AccountTagConfigFunctionRole.Arn + AccountRegionConfigFunctionRole: + Type: "AWS::IAM::Role" + Properties: + AssumeRolePolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: "Allow" + Principal: + Service: + - lambda.amazonaws.com + Action: "sts:AssumeRole" + Path: "/aws-deployment-framework/account-management/" + Policies: + - PolicyName: "adf-lambda-account-region-resource-policy" + PolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Action: + - "account:ListRegions" + - "account:EnableRegion" + - "sts:GetCallerIdentity" + Resource: "*" + - Effect: Allow + Action: ssm:GetParameter + Resource: + - !Sub "arn:${AWS::Partition}:ssm:${AWS::Region}:${AWS::AccountId}:parameter/target_regions" + + AccountRegionConfigFunction: + Type: 'AWS::Serverless::Function' + Properties: + Handler: configure_account_regions.lambda_handler + Description: "ADF Lambda Function - Account region Configuration" + CodeUri: lambda_codebase/account_processing + Architectures: + - arm64 + Tracing: Active + Layers: + - !Ref LambdaLayerVersion + Environment: + Variables: + MASTER_ACCOUNT_ID: !Ref AWS::AccountId + ORGANIZATION_ID: !GetAtt Organization.OrganizationId + ADF_VERSION: !FindInMap ['Metadata', 'ADF', 'Version'] + ADF_LOG_LEVEL: !Ref LogLevel + FunctionName: AccountRegionConfigurationFunction + Role: !GetAtt AccountRegionConfigFunctionRole.Arn + AccountOUConfigFunction: Type: 'AWS::Serverless::Function' Properties: @@ -668,7 +718,7 @@ Resources: "Next": "CreateAccount" } ], - "Default": "ConfigureAccountAlias" + "Default": "ConfigureAccountRegions" }, "ConfigureAccountAlias": { "Type": "Task", @@ -745,7 +795,41 @@ Resources: "MaxAttempts": 6 } ], - "Next": "ConfigureAccountAlias" + "Next": "ConfigureAccountRegions" + }, + "ConfigureAccountRegions": { + "Type": "Task", + "Resource": "${AccountRegionConfigFunction.Arn}", + "Retry": [ + { + "ErrorEquals": [ + "Lambda.ServiceException", + "Lambda.AWSLambdaException", + "Lambda.SdkClientException", + "Lambda.TooManyRequestsException" + ], + "IntervalSeconds": 2, + "MaxAttempts": 6, + "BackoffRate": 2 + } + ], + "Next": "AreRegionsConfigured" + }, + "AreRegionsConfigured": { + "Type": "Choice", + "Choices": [ + { + "Variable": "$.all_regions_enabled", + "BooleanEquals": true, + "Next": "ConfigureAccountAlias" + } + ], + "Default": "Wait 15 seconds" + }, + "Wait 15 seconds": { + "Type": "Wait", + "Seconds": 15, + "Next": "ConfigureAccountRegions" }, "ConfigureAccountTags": { "Type": "Task", @@ -1301,7 +1385,15 @@ Resources: python: 3.11 pre_build: commands: - - pip install -r adf-build/requirements.txt -r adf-build/requirements-dev.txt -r adf-build/shared/helpers/requirements.txt --quiet + - >- + pip install + -r adf-build/requirements.txt + -r adf-build/requirements-dev.txt + -r adf-build/shared/requirements.txt + -r adf-build/shared/requirements-dev.txt + -r adf-build/shared/helpers/requirements.txt + -r adf-build/shared/helpers/requirements-dev.txt + --quiet - pytest -vvv build: commands: