Deep-copy proto state to prevent concurrent modification

[jacksontj]

Backport of #6707 to the 1.11.x series; as this impacts 1.11.x and 1.12.x (which has the fix) requires some potentially non-trivial changes to upgrade. IMO this should be merged and a new bugfix release cut. On my clusters protokube (per master) is hitting this every ~3 minutes -- which is causing a lot of excessive churn and cloud-provider calls.

Fix #6635

[k8s-ci-robot]

Welcome @jacksontj!

It looks like this is your first PR to kubernetes/kops 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kops has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Keywords which can automatically close issues and at(@) mentions are not allowed in commit messages.

The list of commits with invalid commit messages:

• 6530f30 Deep-copy proto state to prevent concurrent modification

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[k8s-ci-robot]

Hi @jacksontj. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jacksontj
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: kashifsaadat

If they are not already assigned, you can assign the PR to them by writing /assign @kashifsaadat in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rdrgmnzs]

@jacksontj is there anything preventing you from using kops 1.12 since is already has the fix?

[jacksontj]

TLDR; 1.11 I can update with no impact, 1.12 requires master impact

@rdrgmnzs as mentioned in the original post, 1.12 requires potentially destructive changes for etcd -- and I would rather not make a major version change in a rush. This is a trivial fix (which applies cleanly) so I'd much rather have this bugfix applied to all my clusters (since its a small change) then work on planning the 1.12 upgrade (which requires master impact).

[rifelpet]

@jacksontj The etcd-manager portion of the 1.12 upgrade only applies to upgrading to Kubernetes 1.12. You could still run Kops 1.12 on your Kubernetes 1.11 cluster. You would pick up this fix and the rolling-update would not be destructive.

[jacksontj]

I'll look into upgrading to 1.12, but I'm not sure why the pushback to backport a bugfix to a still-supported release of kops?

[jacksontj]

@rifelpet I have tried the 1.12 upgrade and I'm running into issues with etcd. I do still plan on working through these, but this is not becoming a much more involved upgrade than a bugfix-- can we just merge this backport and cut a bugfix release?

[jacksontj]

My 1.12 upgrade is now completed, there were a few changes required in my config generation stuff (as we're linking with kops); so I personally am past this problem, but I still think this is worthy of a backport if 1.11 is still supported (which it seems to be based on the README).

[jacksontj]

I have actually opened #7134 as this patch doesn't resolve the issue, IMO once there is a fix it should be backported -- but it seems that this patch wasn't sufficient.

[justinsb]

Thanks for #7134 - sorry I got this wrong in the first place. We can think about backporting that to kops 1.11, but I do think users should be on kops 1.12 instead.

I'm going to close this particular PR though, as you implemented a better fix!