kubernetes.io/cluster/<name>=shared tag being added to VPC that isn't managed by kops

[kashook]

We use kops to create Kubernetes clusters in AWS in a preexisting VPC that is maintained outside of kops by a separate Terraform (but we let kops create its own subnets in the VPC). I recently started using kops 1.8 and noticed that kops has started adding the kubernetes.io/cluster/<name>=shared tag to the VPC even though kops is not managing the VPC. Since Terraform doesn't know anything about the tag, it wants to remove it. Up until this point kops had not been editing the VPC as far as I could tell (although the plan erroneously showed that it was going to rename it). Is the kubernetes.io/cluster/<name>=shared tag actually required on the VPC when kops isn't actually making the VPC? Our choices seem to be to let the terraform go ahead and remove the tags that kops added (but I don't know if this will cause something bad to happen), or change the terraform to have the tags, even though that terraform's job is just to make the VPC (and a few other shared pieces of infrastructure) and isn't concerned with other "downstream" processes that happen to be creating other infrastructure in that VPC. I think this pull request may be the one that caused the tag to start showing up.

[yoz2326]

We have a similar setup (as in VPC created and managed via terraform and kops creates the cluster). It is better to use the new set of tags going forward.

In our case, to reconcile the two tools, we create the tags in terraform and use something similar to the code below:

resource "aws_subnet" "scope" {
.........

  tags = "${merge(
    map("VPC", "${var.vpc_name}"),
    map("Name", "${var.vpc_name}-${var.tier}-${var.availability_zone}"),
    map("kubernetes.io/cluster/${var.KubernetesCluster_name}", "shared"),
    map("kubernetes.io/role/${var.elb_scope}", "1")
  )}"
}

[chrislovecnm]

/assign @justinsb

[bdashrad]

This bit us, and is quite frustrating. I don't want to have to change all of our resource tagging. Do we gain anything by tagging the shared vpc?

[krogon]

I can imagine the situation that VPC is not created by terraform, but using kops.
Now you want to deploy second kops cluster within same, existing VPC.

If there isn't "shared" tag delete command from first cluster will try to delete VPC and would fail as there are resources of second cluster.

[bdashrad]

My assumption would be that kops should not try to delete a resource it didn't explicitly create. I'd rather see VPCs that it creates tagged as safe to modify or delete

[krogon]

Once again. Cluster1 creates VPC-XYZ and owns it. Cluster2 uses existing, same VPC-XYZ. Without any "shared" tag deletion of cluster1 will try to destroy this VPC. Actually it will survive due to other resources dependencies.

I pointed out edge case where this could be a issue. But I see your point, that's the problem using two tools managing same resource with separate "state" files.

[chrislovecnm]

@keiths-osc or can someone give us instructions how to recreate this? @krogon's use case is where you have two k8s clusters in the same vpc, and then you have some challenges with the shared or owned tags. If you have two k8s clusters I recommend you create the VPC separately.

[kashook]

@chrislovecnm, in my use case the VPC was not created by kops, but by a separate terraform. Starting with kops 1.8+, each cluster I bring up in the VPC adds a kubernetes.io/cluster/<name>=shared tag to the VPC, which makes the plan for the VPC terraform show that it wants to remove the tags. The steps to reproduce, roughly, are:

1. Use terraform to create a VPC. Note that the VPC does not have any kubernetes.io/cluster/<name>=shared tags
2. Create a kops cluster, but tell kops to use the id of the existing VPC that was created by your VPC terraform. Notice that kops has added a kubernetes.io/cluster/<name>=shared tag to the VPC
3. Run terraform plan for your VPC terraform. Terraform wants to remove the tag because it didn't create it.

(I can give more specific details if needed).

As one user suggested earlier, I could just put the tags in the terraform. However, I'd rather not have to do that, unless there is some functionality in Kubernetes that actually depends on that tag being there that I am not aware of. Before 1.8, kops did not edit the VPC if it didn't make it (even though it erroneously says it's going to rename it in the plan).

[madAndroid]

Just to add to the discussion here, in case others encounter the same thing we have: when managing the existing VPC with Terraform, if a module uses the vpc data source, Terraform is not able to parse the tag that Kops is creating, leading to this error:

* module.example.module.exmple.aws_elasticache_parameter_group.redis_parameter_group: At column 87, line 1: map "data.aws_vpc.vpc.tags" does not have homogenous types. found TypeString and then TypeMap in:

${replace(format("%.255s", lower(replace("tf-redis-${var.name}-${data.aws_vpc.vpc.tags["Name"]}", "_", "-"))), "/\\s/", "-")}

The only way to fix this is by removing the tag that Kops creates, before running Terraform ... I'm glad to see that Kops will no longer attempt to tag the VPC!

[Nuru]

Without this tag, we have no way of identifying (via AWS or Terraform or the monitoring tools that rely on AWS tags) what VPC the kops cluster is using when it is using a shared VPC. While it would be nice if Terraform provided a very easy or even a default way to not manage tags it did not create, you can immediately prevent Terraform form removing the cluster tag by adding

lifecycle {
    # Ignore tags added by kops or kubernetes
    ignore_changes = ["tags.%", "tags.kubernetes"]
  }

to the resource definition. I believe this is a much better solution, because it preserves information and treats tags like other resources with respect to the rule that you should not delete resources you did not create.