Upgrade to v1.6.2 failing with Dependencies stack failure for external deployment

[SennaSemakula]

Describe the bug

Upgrade from v1.5.2 to v1.6.2 failing for external deployment.

This is due to the AWSAccelerator-DependenciesStack failing to create the IAM role AWSAccelerator-DiagnosticsPackAccessRole in other accounts. This is attributed to the trust relationship policy incorrectly referencing an ARN that containers the AcceleratorPrefix but not the AcceleratorQualifier stated in the template parameters. It seems that when you use aws-accelerator as the qualfiier, the diagnostic pack incorrectly assumes this is the default value and fallbacks to using the AcceleratorPrefix instead. If you were to use another value such as aws-accelerator-sandbox for the qualfiier, this works as expected.

https://github.com/awslabs/landing-zone-accelerator-on-aws/blob/main/source/packages/%40aws-accelerator/accelerator/lib/stacks/dependencies-stack/diagnostics-pack.ts#L47 needs to be fixed as aws-accelerator is a valid qualifier and should take precedence instead falling back to AcceleratorPrefix.

To Reproduce

1. Clone https://github.com/awslabs/landing-zone-accelerator-on-aws/releases/tag/v1.6.2
2. Synthesize the installer template with external deployment option (follow this guide: https://docs.aws.amazon.com/solutions/latest/landing-zone-accelerator-on-aws/external-pipeline-deployment.html)
3. Replace the template from the AWSAccelerator-InstallerStack CloudFormation stack ensuring these parameters are set:

AcceleratorPrefix = AWSAccelerator
AcceleratorQualifier = aws-accelerator
ExistingConfigRepositoryBranchName = main
ExistingConfigRepositoryName=aws-accelerator-config
RepositoryBranchName=release/v1.6.2
UseExistingConfigRepo=Yes
ManagementAccountRoleName=AcceleratorPipelineDeploymentRole
EnableDiagnosticsPack=Yes

4. Update the template and the aws-accelerator-installer codepipeline should kick off

5. After the installer pipeline finishes it will kick off the downstream pipeline: aws-accelerator-pipeline

6. The pipeline will fail at the logging stage with the following error:

AWSAccelerator-DependenciesStack-540580036166-us-east-1 \| 1/5 \| 2:26:22 PM \| CREATE_FAILED        \| AWS::IAM::Role      \| DiagnosticsPackAssumeRole (DiagnosticsPackAssumeRole98643D9F) Resource handler returned message: "Invalid principal in policy: "AWS":"arn:aws:iam::356031120245:role/AWSAccelerator-DiagnosticsPackLambdaRole" (Service: Iam, Status Code: 400, Request ID: 4f7e3414-9b6b-49fa-8d0a-e3986c3a88bb)" (RequestToken: 0a753822-70bf-035a-5053-9c1b88cc9be1, HandlerErrorCode: InvalidRequest)
--
151 | new Role (/codebuild/output/src3479/src/s3/00/source/node_modules/aws-cdk-lib/aws-iam/lib/role.js:1:5794)
152 | \_ DiagnosticsPack.createDiagnosticsPackAssumeRole (/codebuild/output/src3479/src/s3/00/source/packages/@aws-accelerator/accelerator/lib/stacks/dependencies-stack/diagnostics-pack.ts:43:20)
153 | \_ new DiagnosticsPack (/codebuild/output/src3479/src/s3/00/source/packages/@aws-accelerator/accelerator/lib/stacks/dependencies-stack/diagnostics-pack.ts:20:10)
154 | \_ new DependenciesStack (/codebuild/output/src3479/src/s3/00/source/packages/@aws-accelerator/accelerator/lib/stacks/dependencies-stack/dependencies-stack.ts:49:7)
155 | \_ createKeyDependencyStacks (/codebuild/output/src3479/src/s3/00/source/packages/@aws-accelerator/accelerator/utils/stack-utils.ts:621:29)
156 | \_ createMultiAccountMultiRegionStacks (/codebuild/output/src3479/src/s3/00/source/packages/@aws-accelerator/accelerator/bin/app.ts:161:32)
157 | \_ main (/codebuild/output/src3479/src/s3/00/source/packages/@aws-accelerator/accelerator/bin/app.ts:243:5)
158 | \_ processTicksAndRejections (node:internal/process/task_queues:96:5)
159 | \_ async /codebuild/output/src3479/src/s3/00/source/packages/@aws-accelerator/accelerator/bin/app.ts:251:5
160 | AWSAccelerator-DependenciesStack-540580036166-us-east-1 \| 1/5 \| 2:26:22 PM \| UPDATE_ROLLBACK_IN_P \| AWS::CloudFormation::Stack \| AWSAccelerator-DependenciesStack-540580036166-us-east-1 The following resource(s) failed to create: [DiagnosticsPackAssumeRole98643D9F].

Expected behavior
With EnableDiagnosticsPack set to 'Yes' in the installer template, the aws-accelerator-pipeline pipeline should pass and create the following IAM role AWSAccelerator-DiagnosticsPackAccessRole in all accounts. This IAM role would have the correct ARN in the trust policy that references the qualifier used in the parameters e.g.

Correct:
arn:aws:iam::<account_id>:role/<AcceleratorQualifier>-DiagnosticsPackLambdaRole
Incorrect:
arn:aws:iam::<account_id>:role/<AcceleratorPrefix>-DiagnosticsPackLambdaRole

Please complete the following information about the solution:

• Version: v1.6.2

To get the version of the solution, you can look at the description of the created AWS CloudFormation stack used to install the LZA (AWSAccelerator-InstallerStack). For example, "(SO0199) Landing Zone Accelerator on AWS. Version 1.5.1.". If the description does not contain the version information, you can look at the Parameters of the stack for the RepositoryBranchName as that should contain the version number.

• Region: us-east-1
• Was the solution modified from the version published on this repository? No.
• If the answer to the previous question was yes, are the changes available on GitHub?
• Have you checked your service quotas for the services this solution uses? Yes.
• Were there any errors in the CloudWatch Logs? No.

Screenshots

Additional context

If you set the qualifier to any value other than aws-accelerator this will work as expected

[erwaxler]

Hi @SennaSemakula , thanks for reaching out to the Landing Zone Accelerator team.

Thank you for the detailed write-up, I have observed this behavior in my environment as well. The team has implemented a fix that will be included in our next major release, I'll keep the issue open until it is available.

In the meantime, you can workaround this behavior with the following steps:

1. Update the AWSAccelerator-InstallerStack and set the parameter EnableDiagnosticsPack to no.
2. Manually create an IAM role named AWSAccelerator-DiagnosticsPackLambdaRole in the pipeline account.
   a. Note - the attached policies do not have to match the other DiagnosticsPack role, a role with that name simply has to exist.
3. Rerun the installer pipeline.

Thank you for raising this issue and supporting the Landing Zone Accelerator!

[SennaSemakula]

Hi @erwaxler,

Thanks for the prompt response!

Setting EnableDiagnosticsPack to No seems like there is an existing bug. We haven't tried it but can you first confirm we won't run into the following issue: #379

[erwaxler]

Hi @SennaSemakula , I can't make promises since I don't know the full details of your environment, but I did not experience that in my environment using v1.6.2.

You can also try manually creating the role without changing the EnableDiagnosticsPack parameter first, then retrying the failing pipeline stage. I did both, but based on your error message you may only need a role with that name to exist in order to move forward.

[SennaSemakula]

@erwaxler great, thanks. Could you give us an ETA on the next release as we use LZA in production.

[erwaxler]

No problem! Unfortunately I cannot provide a specific timeline for future releases.

Thank you for your support and interest of the LZA solution! I will leave this issue open should you have any follow-ups for the team, and we will update you when the feature becomes available.

[islamgohar]

workaround worked for me, thanks @erwaxler
note:

• a very straight forward 1.6.3 deployment -with Diagnostics Pack disabled- failed for me.
• I had to disable termination protection on AWSAccelerator-DependenciesStack on both audit and logArchive accounts for the workaround to work.

[SennaSemakula]

Hi @SennaSemakula , thanks for reaching out to the Landing Zone Accelerator team.

Thank you for the detailed write-up, I have observed this behavior in my environment as well. The team has implemented a fix that will be included in our next major release, I'll keep the issue open until it is available.

In the meantime, you can workaround this behavior with the following steps:

1. Update the AWSAccelerator-InstallerStack and set the parameter EnableDiagnosticsPack to no.
2. Manually create an IAM role named AWSAccelerator-DiagnosticsPackLambdaRole in the pipeline account.
   a. Note - the attached policies do not have to match the other DiagnosticsPack role, a role with that name simply has to exist.
3. Rerun the installer pipeline.

Thank you for raising this issue and supporting the Landing Zone Accelerator!

Workaround worked on v1.6.2. We manually created the role before updating the installer stack to prevent retrying the pipeline and the pipeline succeeded. Thanks!

[erwaxler]

Hi @SennaSemakula , this has been included in our v1.7.0 release so I will be closing this issue. Thanks for supporting the Landing Zone Accelerator!