feat: Implement support for KMS arguments #288

mkilchhofer · 2023-01-06T16:44:45Z

Describe your changes

This implements Add Key Management Service (KMS) etcd encryption to an Azure Kubernetes Service (AKS) cluster.

Feature is not yet available in azurerm provider, but also opened a PR over there:

azurerm_kubernetes_cluster - Support for KMS arguments hashicorp/terraform-provider-azurerm#19893

Issue number

-

Checklist before requesting a review

The pr title can be used to describe what this pr did in CHANGELOG.md file
I have executed pre-commit on my machine
I have passed pr-check on my machine

Thanks for your cooperation!

lonegunmanb

Thanks @mkilchhofer for opening this pr and submitting a new resource pr to terraform-azurerm-provider's repo, I've left some comments on provider's pr, once the pr has been merged into the provider, we can continue the code review for this pr.

Btw the review comments for your provider's pr are my personal suggestions, there's no guarantee that your pr would be accepted by HashiCorp if you accepted my suggestion.

lonegunmanb · 2023-01-08T14:26:56Z

main.tf

@@ -136,6 +136,16 @@ resource "azurerm_kubernetes_cluster" "main" {
      subnet_id    = var.ingress_application_gateway_subnet_id
    }
  }
+  dynamic "key_vault_kms" {
+    for_each = var.key_vault_kms_enabled ? ["key_vault_kms"] : []


Since we could turn enabled to false meanwhile keeping this key_vault_kms, I would recommend the following toggle expression:

for_each = var.key_vault_kms_enabled != null ? ["key_vault_kms"] : []

refactored this anyways (source was the upstream implementation)

mkilchhofer · 2023-01-20T09:54:06Z

PR is now updated after the upstream provider PR got merged AND released ;-)

@lonegunmanb can you please review again?

lonegunmanb

Thanks @mkilchhofer for updating this pr, some issues need to be solved.

It would be nice if we could have an new example to demonstrate this new feature. We have already created a KeyValut in startup example, but for disk encryption key management for now, would you please estimate whether if we can make that KeyVault work for this new feature? We might need to update azurerm provider's restriction in providers.tf file in startup folder. Or we can just crate a new example folder to do so, your call.

Again, thanks for your contribution!

variables.tf

main.tf

mkilchhofer · 2023-01-23T13:18:07Z

I cannot test the examples as we @swisspost have no subscriptions which allows usage of public IPs.
Can someone test that for us?

zioproto · 2023-01-23T14:00:39Z

@mkilchhofer I can do that ! thanks

zioproto · 2023-01-23T14:18:11Z

@mkilchhofer could you please run the pre-commit step on your local machine ?:

docker run --rm -v $(pwd):/src -w /src mcr.microsoft.com/azterraform:latest make pre-commit

You patched the example and you have duplicated definitions for identity_type and identity_ids. See the error:

Error: Attribute redefined
│
│   on examples/named_cluster/main.tf line 88, in module "aks_cluster_name":
│   88:   identity_type         = "UserAssigned"
│
│ The argument "identity_type" was already set at
│ examples/named_cluster/main.tf:66,3-16. Each argument may be set only once.
╵

╷
│ Error: Attribute redefined
│
│   on examples/named_cluster/main.tf line 89, in module "aks_cluster_name":
│   89:   identity_ids          = [azurerm_user_assigned_identity.test.id]
│
│ The argument "identity_ids" was already set at
│ examples/named_cluster/main.tf:65,3-15. Each argument may be set only once.

- nullable=false on `kms_enabled` - Validate kms_enabled and UserAssigned identity

zioproto

Could you patch only the named_cluster example ?

We dont need to the KMS in the others, it would also require more work because the needed UserAssigned identity_type is not set in the others. Thanks

examples/named_cluster/main.tf

zioproto · 2023-01-24T13:15:29Z

commit 7012178 is now passing both pre-commit and pr-check steps.

@lonegunmanb could you please review and run the e2e tests ? thanks

.checkov_config.yaml

variables.tf

examples/named_cluster/kms.tf

mkilchhofer · 2023-01-27T10:09:01Z

@lonegunmanb can you trigger E2E again? 😇

lonegunmanb

Thanks @mkilchhofer for this update, we've met some issues when we ran the test.

We've added network_acl on the KeyVault to restrict the source IP which can access this Key Vault, but the Aks's public IP is unpredictable so the original code would meet a 403 error.

We need config the aks to access the KeyVault via private network as this document described.

examples/named_cluster/kms.tf

lonegunmanb · 2023-01-28T09:49:32Z

examples/named_cluster/main.tf

@@ -77,4 +77,12 @@ module "aks_cluster_name" {
  rbac_aad                          = true
  rbac_aad_managed                  = true
  role_based_access_control_enabled = true
+


We need the following line below line 31:

service_endpoints = ["Microsoft.KeyVault"]

Service endpoint is not needed (at least in our scenario). I think its only needed if some part inside the VNET wants to access the keyvault.

examples/named_cluster/main.tf

examples/named_cluster/kms.tf

lonegunmanb

Thanks @mkilchhofer, LGTM! 🚀

mkilchhofer · 2023-01-29T08:30:52Z

Thanks alot for your patience with me 👍

mkilchhofer force-pushed the feature/secrets_kms_encryption branch 2 times, most recently from 06b70fa to fc2b23f Compare January 8, 2023 12:10

lonegunmanb requested changes Jan 8, 2023

View reviewed changes

mkilchhofer force-pushed the feature/secrets_kms_encryption branch 2 times, most recently from a8c2e97 to 1eef1ef Compare January 20, 2023 09:52

mkilchhofer marked this pull request as ready for review January 20, 2023 09:53

mkilchhofer requested a review from lonegunmanb January 20, 2023 09:54

feat: Implement support for KMS arguments

25c45a7

mkilchhofer force-pushed the feature/secrets_kms_encryption branch from 1eef1ef to 25c45a7 Compare January 20, 2023 09:59

lonegunmanb requested changes Jan 21, 2023

View reviewed changes

variables.tf Show resolved Hide resolved

variables.tf Outdated Show resolved Hide resolved

zioproto requested changes Jan 23, 2023

View reviewed changes

main.tf Show resolved Hide resolved

mkilchhofer force-pushed the feature/secrets_kms_encryption branch from 56eb644 to 2058adc Compare January 23, 2023 12:50

chore: Apply changes from code review

0d2123c

- nullable=false on `kms_enabled` - Validate kms_enabled and UserAssigned identity

mkilchhofer force-pushed the feature/secrets_kms_encryption branch from 54029b0 to c0cde66 Compare January 23, 2023 14:56

zioproto requested changes Jan 23, 2023

View reviewed changes

examples/named_cluster/main.tf Outdated Show resolved Hide resolved

examples/named_cluster/main.tf Outdated Show resolved Hide resolved

chore: Update examples with KMS

eda1086

mkilchhofer force-pushed the feature/secrets_kms_encryption branch from c0cde66 to eda1086 Compare January 23, 2023 19:02

mkilchhofer requested review from zioproto and lonegunmanb and removed request for zioproto January 24, 2023 08:08

mkilchhofer added 2 commits January 24, 2023 09:48

chore: Set azurerm minimal version to 3.40

2bb6f10

chore: Fix TF linter errors on named_cluster example

f61f35c

mkilchhofer force-pushed the feature/secrets_kms_encryption branch from 56d79c3 to f61f35c Compare January 24, 2023 09:13

chore: Fulfill checkov checks

7012178

zioproto reviewed Jan 24, 2023

View reviewed changes

.checkov_config.yaml Show resolved Hide resolved

fix: Fixup precondition check for UserAssigned identity type

07f427e

mkilchhofer had a problem deploying to acctests January 26, 2023 14:04 — with GitHub Actions Error

mkilchhofer had a problem deploying to acctests January 27, 2023 03:58 — with GitHub Actions Failure

lonegunmanb requested changes Jan 27, 2023

View reviewed changes

variables.tf Outdated Show resolved Hide resolved

examples/named_cluster/kms.tf Outdated Show resolved Hide resolved

mkilchhofer added 2 commits January 27, 2023 10:26

fix: Add depends_on for KV access policy in examples

c4a8bb9

chore: Use "Public" as default instead of null

05c1b8e

mkilchhofer had a problem deploying to acctests January 28, 2023 02:19 — with GitHub Actions Failure

lonegunmanb requested changes Jan 28, 2023

View reviewed changes

fix: Use private endpoint for vault connection

f13b78a

mkilchhofer temporarily deployed to acctests January 28, 2023 14:36 — with GitHub Actions Inactive

lonegunmanb approved these changes Jan 29, 2023

View reviewed changes

lonegunmanb merged commit 5443398 into Azure:master Jan 29, 2023

mkilchhofer deleted the feature/secrets_kms_encryption branch January 30, 2023 10:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Implement support for KMS arguments #288

feat: Implement support for KMS arguments #288

mkilchhofer commented Jan 6, 2023 •

edited

Loading

lonegunmanb left a comment •

edited

Loading

lonegunmanb Jan 8, 2023

mkilchhofer Jan 20, 2023 •

edited

Loading

mkilchhofer commented Jan 20, 2023

lonegunmanb left a comment

mkilchhofer commented Jan 23, 2023

zioproto commented Jan 23, 2023

zioproto commented Jan 23, 2023

zioproto left a comment

zioproto commented Jan 24, 2023

mkilchhofer commented Jan 27, 2023

lonegunmanb left a comment

lonegunmanb Jan 28, 2023

mkilchhofer Jan 28, 2023

lonegunmanb left a comment

mkilchhofer commented Jan 29, 2023

feat: Implement support for KMS arguments #288

feat: Implement support for KMS arguments #288

Conversation

mkilchhofer commented Jan 6, 2023 • edited Loading

Describe your changes

Issue number

Checklist before requesting a review

lonegunmanb left a comment • edited Loading

Choose a reason for hiding this comment

lonegunmanb Jan 8, 2023

Choose a reason for hiding this comment

mkilchhofer Jan 20, 2023 • edited Loading

Choose a reason for hiding this comment

mkilchhofer commented Jan 20, 2023

lonegunmanb left a comment

Choose a reason for hiding this comment

mkilchhofer commented Jan 23, 2023

zioproto commented Jan 23, 2023

zioproto commented Jan 23, 2023

zioproto left a comment

Choose a reason for hiding this comment

zioproto commented Jan 24, 2023

mkilchhofer commented Jan 27, 2023

lonegunmanb left a comment

Choose a reason for hiding this comment

lonegunmanb Jan 28, 2023

Choose a reason for hiding this comment

mkilchhofer Jan 28, 2023

Choose a reason for hiding this comment

lonegunmanb left a comment

Choose a reason for hiding this comment

mkilchhofer commented Jan 29, 2023

mkilchhofer commented Jan 6, 2023 •

edited

Loading

lonegunmanb left a comment •

edited

Loading

mkilchhofer Jan 20, 2023 •

edited

Loading