feat(TF>=1.1)!: Configure ASM management mode

[ferrarimarco]

This PR implements the ability of configuring ASM management mode.

Without this, users have to manually configure the feature membership.

[ferrarimarco]

Ah! But we configure the CPR already, so this may not actually be needed...

[ericyz]

/gcbrun

[ferrarimarco]

Also, it looks like that the module.cpr and kubernetes_config_map.asm_options aren't necessary anymore because the fleet API takes care of that, and actually conflict with the Managed ASM control plane installation. Worth removing in this PR, or a separate one?

[ferrarimarco]

PS: the build failure doesn't seem to be related to this PR.

[apeabody]

/gcbrun

[apeabody]

/gcbrun

[apeabody]

Thanks for the pull request @ferrarimarco

From the linter

Checking submodule's files generation
diff -r '--exclude=.terraform' '--exclude=.kitchen' '--exclude=.git' /workspace/modules/asm/README.md /tmp/tmp.eM1y9hxVzy/workspace/modules/asm/README.md
62c62
< | enable\_vpc\_sc | Determines whether to enable VPC-SC for this ASM installation. For more information read <https://cloud.google.com/service-mesh/docs/managed/vpc-sc> | `bool` | `false` | no |
---
> | enable\_vpc\_sc | Determines whether to enable VPC-SC for this ASM installation. For more information read https://cloud.google.com/service-mesh/docs/managed/vpc-sc | `bool` | `false` | no |
65c65
< | mesh\_management | ASM management mode. For more information, see the [gke_hub_feature_membership resource documentation](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/gke_hub_feature_membership#nested_mesh) | `string` | `""` | no |
---
> | mesh\_management | ASM Management mode. | `string` | `""` | no |
Error: submodule's files generation has not been run, please run the
'make build' command and commit changes

Checking for documentation generation
diff -r '--exclude=.terraform' '--exclude=.kitchen' '--exclude=autogen' '--exclude=*.tfvars' '--exclude=*metadata.yaml' /workspace/modules/asm/README.md /tmp/tmp.3rgofwWMtd/generate_docs/workspace/modules/asm/README.md
62c62
< | enable\_vpc\_sc | Determines whether to enable VPC-SC for this ASM installation. For more information read <https://cloud.google.com/service-mesh/docs/managed/vpc-sc> | `bool` | `false` | no |
---
> | enable\_vpc\_sc | Determines whether to enable VPC-SC for this ASM installation. For more information read https://cloud.google.com/service-mesh/docs/managed/vpc-sc | `bool` | `false` | no |
65c65
< | mesh\_management | ASM management mode. For more information, see the [gke_hub_feature_membership resource documentation](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/gke_hub_feature_membership#nested_mesh) | `string` | `""` | no |
---
> | mesh\_management | ASM Management mode. | `string` | `""` | no |
Error: Documentation generation has not been run, please run the
'make docker_generate_docs' command and commit the above changes.

[ferrarimarco]

@apeabody thanks for the update. I'll look into that.

[ferrarimarco]

@apeabody hopefully fixed!

[apeabody]

@apeabody hopefully fixed!

Thanks @ferrarimarco - With the addition of count to the module, you will also need to add the index for the output value:

│ Error: Unsupported attribute
│ 
│   on ../../../modules/asm/outputs.tf line 23, in output "wait":
│   23:   value       = module.cpr.wait
│     ├────────────────
│     │ module.cpr is a list of object
│ 
│ Can't access attributes on a list of objects. Did you mean to access
│ attribute "wait" for a specific element of the list, or across all elements
│ of the list?

[ferrarimarco]

@apeabody thanks for your review.

I've modified the wait output variable initialization to fix the issue reported in #1702 (comment).

Also, the module now accordingly sets its value considering both MANAGEMENT_AUTOMATIC and MANAGEMENT_MANUAL (or unset management mode).

[apeabody]

/gcbrun

[apeabody]

/gcbrun

[ferrarimarco]

Build failures seem related to quota issues.

[apeabody]

/gcbrun

[ferrarimarco]

This failure seems to be related to an unsupported GKE version. I don't think it's related to this PR :)

[apeabody]

Also, it looks like that the module.cpr and kubernetes_config_map.asm_options aren't necessary anymore because the fleet API takes care of that, and actually conflict with the Managed ASM control plane installation. Worth removing in this PR, or a separate one?

Hi @ferrarimarco - Is there a recommend migration path? While reviewing this PR its clear that supporting both adds extra complexity. Ideally we would want to minimze wrapping imperative commands like in module.cpr.

[ferrarimarco]

Currently, Anthos Service Mesh supports the following installation methods:

• Managed ASM: managed control plane that runs outside the cluster.
  • using Fleet API: fully automated, recommended by ASM docs.
  • using asmcli.
• In-cluster ASM: control plan runs inside the cluster. Install using asmcli.

In all cases, there's no explicit creation of ControlPlaneRevision resources. Either the Fleet API does it, or asmcli takes care of creating it.

My opinion is that we should:

• Remove the module.cpr resource because the Fleet API and asmcli take care of creating it.
• Remove the kubernetes_namespace.system resource because the Fleet API and asmcli take care of creating it.
• Clarify that we install the "managed ASM control plane" version using the Fleet API.
• Refactor optional features according to Enable optional features on managed ASM. This probably requires dropping unsupported features, such as var.enable_cni. See the update.

Installation methods that involve using asmcli will likely result in having to wrap calls to asmcli using null_resources, which are hard to maintain and to implement with the right triggers. That's why I suggest sticking with provisioning and configuring managed ASM using the Fleet API only.

I'm happy to talk about this over Meet if you prefer :)

/cc @jbrook

UPDATE: I think we should also drop the dependency from the Kubernetes provider because we wouldn't need to connect to individual clusters in the fleet and avoid configuring optional features at all Enable optional features on managed ASM. As pointed out by folks I aligned with, it may be unlikely that we want users to manage individual objects inside clusters with Terraform. We might want to defer optional features configuration to a later PR, provided that we see demand.

UPDATE 2: By dropping the dependency from the Kubernetes provider, we'd need to find an alternative way to check if the ControlPlaneRevision is there, which we use as a proxy to check if ASM mutating web hooks are in place.

[apeabody]

Currently, Anthos Service Mesh supports the following installation methods:

• Managed ASM: managed control plane that runs outside the cluster.

  • using Fleet API: fully automated, recommended by ASM docs.
  • using asmcli.

• In-cluster ASM: control plan runs inside the cluster. Install using asmcli.

In all cases, there's no explicit creation of ControlPlaneRevision resources. Either the Fleet API does it, or asmcli takes care of creating it.

My opinion is that we should:

• Remove the module.cpr resource because the Fleet API and asmcli take care of creating it.
• Remove the kubernetes_namespace.system resource because the Fleet API and asmcli take care of creating it.
• Clarify that we install the "managed ASM control plane" version using the Fleet API.
• Refactor optional features according to Enable optional features on managed ASM. This probably requires dropping unsupported features, such as var.enable_cni.

Installation methods that involve using asmcli will likely result in having to wrap calls to asmcli using null_resources, which are hard to maintain and to implement with the right triggers. That's why I suggest sticking with provisioning and configuring managed ASM using the Fleet API only.

I'm happy to talk about this over Meet if you prefer :)

/cc @jbrook

Thanks @ferrarimarco - Given the scale of the change, let me ping you directly.

[apeabody]

/gcbrun

[apeabody]

Currently, Anthos Service Mesh supports the following installation methods:

• Managed ASM: managed control plane that runs outside the cluster.

  • using Fleet API: fully automated, recommended by ASM docs.
  • using asmcli.

• In-cluster ASM: control plan runs inside the cluster. Install using asmcli.

In all cases, there's no explicit creation of ControlPlaneRevision resources. Either the Fleet API does it, or asmcli takes care of creating it.
My opinion is that we should:

• Remove the module.cpr resource because the Fleet API and asmcli take care of creating it.
• Remove the kubernetes_namespace.system resource because the Fleet API and asmcli take care of creating it.
• Clarify that we install the "managed ASM control plane" version using the Fleet API.
• Refactor optional features according to Enable optional features on managed ASM. This probably requires dropping unsupported features, such as var.enable_cni.

Installation methods that involve using asmcli will likely result in having to wrap calls to asmcli using null_resources, which are hard to maintain and to implement with the right triggers. That's why I suggest sticking with provisioning and configuring managed ASM using the Fleet API only.
I'm happy to talk about this over Meet if you prefer :)
/cc @jbrook

Thanks @ferrarimarco - Given the scale of the change, let me ping you directly.

Hi @ferrarimarco - Did we want to move forward with this change for now? Cheers!

[apeabody]

/gcbrun

[ferrarimarco]

Hi Andrew! I'll align internally and get back to you ASAP. Thanks for following up!

[apeabody]

/gcbrun

[apeabody]

/gcbrun

[apeabody]

Thanks @ferrarimarco!