The stack fails at last stage due to custom key store failing to connect with HSM cluster

[awspankj]

When you choose to create custom key store, the stack gets stuck for more than 1.5 hours and then rolls back. The custom key store gets created but fails to connect with HSM cluster with error- "KMS cannot connect the custom key store to its CloudHSM cluster. Error code: USER_NOT_FOUND". I assume 'kmsuser' is not getting configured correctly.

[ckamps]

@awspankj thanks for the report. We're in the process of preparing a heavily refactored version of this automation for publishing here in aws-samples. The refactored version has many enhancements including more robust error handling.

In the meantime, did you get a chance to inspect the cfn-init.log data to potentially better understand the issue? As mentioned in Troubleshooting stack creation.

Selecting the following option during stack creation can help preserve some of the resources so that it's easier to troubleshoot:

Separately, I've also provided you with a pointer to the heavily refactored fork in case you'd like to try that version.

[ckamps]

@awspankj it appears that the kmsuser was created, but at the point that the CloudHSM key store was being connected, the kmuser is in an inconsistent state. i.e. the user is not present on each of the two HSMs in the cluster. The connect operation fails due to the user not being present in all HSMs of the cluster.

I'm investigating why, under some circumstances, the user gets into that state. This failure appears to be a result of enhancing the code to use the cloudhsm-cli package vs the cloudhsm-client package.

While the CloudFormation stack is waiting for the key store to get into the connected state, a workaround is to access the EC2 client, delete, and create again the kmsuser using the the cloudhsm-cli.

[awspankj]

Hi @christopher ***@***.***> Thanks for the update. This is interesting and something new I learned. After the stack was failing in the end, I learned to do it manually with latest cloudhsm CLI commands + console – attached are the steps. Hope to see the github version working again! Meanwhile I will test the fork version. Warm Regards, Pankaj Patil Partner Solutions Architect WWPS Mob- +919967936244 From: Christopher Kampmeier ***@***.***> Reply to: aws-samples/aws-cloudhsm-cloudformation-template ***@***.***> Date: Saturday, 13 May 2023 at 3:56 AM To: aws-samples/aws-cloudhsm-cloudformation-template ***@***.***> Cc: "Patil, Pankaj" ***@***.***>, Mention ***@***.***> Subject: Re: [aws-samples/aws-cloudhsm-cloudformation-template] The stack fails at last stage due to custom key store failing to connect with HSM cluster (Issue #12) @awspankj<https://github.com/awspankj> it appears that the kmsuser was created, but at the point that the CloudHSM key store was being connected, the kmuser is in an inconsistent state. i.e. the user is not present on each of the two HSMs in the cluster. The connect operation fails due to the user not being present in all HSMs of the cluster. I'm investigating why, under some circumstances, the user gets into that state. This failure appears to be a result of enhancing the code to use the cloudhsm-cli package vs the cloudhsm-client package. — Reply to this email directly, view it on GitHub<#12 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AZQPQMO7RRGATCUO75GF4OTXF22H7ANCNFSM6AAAAAAX4WPLZM>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[ckamps]

@awspankj I reverted this repository to the commit prior to introducing use of the cloudhsm-cli package in place of the cloudhsm-client package so that the kmsuser creation is stable. I'll send a note to you once the newly refactored form of the overall automation is published to this repository. In the meantime, you can use the internal fork I referenced separately.

[awspankj]

@christopher ***@***.***> I gave it a try just now: 1. The stack completed (with parameter ‘custom key store’) 2. Cluster is in active state. However, Custom key store is not created. ***@***.*** 1. The Kmsuser is not created ***@***.*** 1. Creation of kmsuser succeeds when done manually: ***@***.*** 1. Now using the kmsuser and anchor certificate stored in secrets manager, custom key store is created from aws console. ***@***.*** Warm Regards, Pankaj Patil Partner Solutions Architect WWPS Mob- +919967936244 From: Christopher Kampmeier ***@***.***> Reply to: aws-samples/aws-cloudhsm-cloudformation-template ***@***.***> Date: Sunday, 14 May 2023 at 9:43 PM To: aws-samples/aws-cloudhsm-cloudformation-template ***@***.***> Cc: "Patil, Pankaj" ***@***.***>, Mention ***@***.***> Subject: Re: [aws-samples/aws-cloudhsm-cloudformation-template] The stack fails at last stage due to custom key store failing to connect with HSM cluster (Issue #12) @awspankj<https://github.com/awspankj> I reverted this repository to the changes prior to introducing use of the cloudhsm-cli package in place of the cloudhsm-client package so that the kmsuser creation is stable. I'll send a note to you once the newly refactored form of the overall automation is published to this repository. In the meantime, you can use the internal fork I referenced separately. — Reply to this email directly, view it on GitHub<#12 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AZQPQMKAUTPRM43US6RW4P3XGEABTANCNFSM6AAAAAAX4WPLZM>. You are receiving this because you were mentioned.Message ID: ***@***.***>