Using existing subnets should not tag them

[dee-kryvenko]

1. What kops version are you running? The command kops version, will display
this information.

1.9.2

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

1.9.9

3. What cloud provider are you using?

AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

Simplest way - create your vpc and subnets using terraform, tag them to your liking, then use kops to create cluster in there, and then try to run terraform apply on top of what you've got.

5. What happened after the commands executed?

k8s tags got deleted from subnet

6. What did you expect to happen?

I expected kops not to touch resources that it is not ow

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

It's not really relevant but I can provide it if needed

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.

9. Anything else do we need to know?

[dee-kryvenko]

I’m not sure what is this tags used for, but as alternative - let user to customize tag query to search for needed resources.

[dee-kryvenko]

FYI - TF has plans to fix this on their side: hashicorp/terraform#6632. Nevertheless, I think it must be revisited if kops doing a right thing here.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[dmildh]

Would be nice if kops could use the provided subnetid instead of tags to avoid these issues.

[seanson]

There is now a kops flag --disable-subnet-tags that allows you to manage the subnet tags out of band.

[dmildh]

@seanson Does that mean we would still need to tag the existing AWS subnet resources with our Terraform setup that is managing the vpc? We specify the subnets: vars in our kops.config already so the configuration already contains the cidrs/types/zones etc.

[seanson]

Yes, subnets should still be tagged otherwise any services that require subnet identification like ELBs will not work.

https://github.com/kubernetes/kops/blob/master/docs/run_in_existing_vpc.md#subnet-tags

[dmildh]

Gotcha, that's what I thought. Since you can already provide all of the information in the kops.config, it would be nice if tags were not required at all for the subnets but assuming that is a Kubernetes requirement not a kops one.

[dee-kryvenko]

Something that I've learned since I've opened this issue is that k8s itself using these tags for various things. So it doesn't sounds like a kops issue, but a k8s issue. Now, the issue is rather big as lies in between of different infrastructure management tools. Such as in my case, it's Terraform (which I use to manage infrastructure such as networking layer) and kubernetes. I had multiple conversations since I posted the issue with various people, and I'm getting mixed opinions. Some people tend to say terraform is not doing a right thing here, since tags supposed to be a separate entity manageable individually from different tools to facilitate service discovery. Me personally, I would argue it's not a terraform fault as they just transparently mimic what an AWS API objects looks like. In AWS - tags is not a separate entity but an attribute to other entities and hence managed the way they managed to facilitate reproducibility, immutability and idempotence. So maybe AWS is at fault here? Just guessing :)

And keep in minds it's not just TF vs k8s... I would imagine similar things could happen with CloudFormation or Ansible on the left and whatever else service-discoverable thing on the right. And also in different cloud providers and virtualization software.

This would be extremely complicated to make everyone sit together and make an RFC once and for all on what the tag is. Or maybe there is one already? Is it a key-value or just a label? Is it a separate entity or an attribute to another entity? And so on and so forth...

To me it sounds unrealistic and I'm not really sure where to move from here. Maybe I'd start from re-opening a similar ticket in k8s issues, and since I already opened similar one in TF I will link them both together.

[dmildh]

@llibicpep Thanks. At this point, I am going to just add the needed tags to my tfvars for the public and private subnets so at least it doesn't try and strip them. If Terraform adds support for wildcards in the ignore_changes option this will help but doesn't appear to be on the map yet.

[dee-kryvenko]

Just to link discussions opened in TF and k8s:
hashicorp/terraform-provider-aws#5974
kubernetes/kubernetes#73190

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.