Update dual stack KEP metadata to include status and PRR fields

[lachie83]

• Add milestone and status fields
• Add PRR related fields in preparation for beta

Signed-off-by: Lachlan Evenson lachlan.evenson@microsoft.com

[lachie83]

Tracking enhancement issue #563

[bridgetkromhout]

Here's how I could reproduce the failure:

$ ./hack/verify-kep-metadata.sh
--- FAIL: TestValidation (0.06s)
    --- FAIL: TestValidation/../../keps/sig-network/563-dual-stack/kep.yaml (0.00s)
        main_test.go:112: PRR approval is required to target milestone v1.21 (stage beta)
        main_test.go:113: For more details about PRR approval see: https://github.com/kubernetes/community/blob/master/sig-architecture/production-readiness.md
        main_test.go:114: To get PRR approval modify appropriately file ../../keps/prod-readiness/sig-network/563.yaml and have this approved by PRR team
FAIL
FAIL	k8s.io/enhancements/cmd/kepval	0.854s
FAIL
$ 

I think this is how we correct this test - update keps/prod-readiness/sig-network/563.yaml (but not with just any name):

$ ./hack/verify-kep-metadata.sh
--- FAIL: TestValidation (0.09s)
    --- FAIL: TestValidation/../../keps/sig-network/563-dual-stack/kep.yaml (0.00s)
        main_test.go:122: PRR approval file ../../keps/prod-readiness/sig-network/563.yaml has an error: error validating PRR approval metadata: invalid beta field: "approver" must be one of (deads2k,johnbelamaric,wojtek-t) but it is a string: thockin
FAIL
FAIL	k8s.io/enhancements/cmd/kepval	0.905s
FAIL
$

We put one of the approved people in this file:

$ cat keps/prod-readiness/sig-network/563.yaml 
kep-number: 563
beta:
  approver: "@wojtek-t"
$

Then this check completes:

$ ./hack/verify-kep-metadata.sh
ok  	k8s.io/enhancements/cmd/kepval	0.661s
$

I'm not sure who's correct to list in this case.

[jeremyrickard]

I don’t believe that Tim is a PRR approver, I believe only @deads2k or @johnbelamaric or @wojtek-are the only PRR approvers currently. Looks like the PRR approver field is empty, I’d recommend adding one of the approvers there.

[bridgetkromhout]

@jeremyrickard I do indeed see that they are the only ones in prod-readiness-approvers. When quickly scanning https://github.com/kubernetes/community/blob/master/sig-architecture/production-readiness.md#submitting-a-kep-for-production-readiness-approval I was looking at the sig-network-leads list in https://github.com/kubernetes/enhancements/blob/master/OWNERS_ALIASES#L57-L60 instead of literally prod-readiness-approvers.

Of course, we'll start with our SIG leads and then get a PRR approval. Thanks!

[lachie83]

I don’t believe that Tim is a PRR approver, I believe only @deads2k or @johnbelamaric or @wojtek-are the only PRR approvers currently. Looks like the PRR approver field is empty, I’d recommend adding one of the approvers there.

I've connected with @johnbelamaric via the prod-readiness channel and he has agreed to review this once ready. I will push a commit to update.

[lachie83]

PRR sections added to KEP here - #2325

[lachie83]

@johnbelamaric this is ready for review.

[dcbw]

I skimmed the checklist and I think it's accurate.
/lgtm
/approve

[johnbelamaric]

Hi. I don't think the latest round of revisions really answers my questions. I think to answer them you need to shift mindset a little. These questions aren't about "how can this specific code fail" but about "what could go wrong now that this change is out there". That is, they really aren't narrowly focused on just your changes, but rather on how those changes interact with the system as a whole. What are the downstream consequences of enabling this feature?

Really put yourself in the shoes of someone that is responsible for 10s or 100s of thousands of workloads running on a fleet of clusters, and ask yourself: Am I willing to enable this feature? If I do, how can I make sure that I haven't broken the cluster control plane? How can I make sure I haven't broken user workloads? Is there any planning I should do before I enable it? How can I go home and not worry about getting paged? What do I have to check, what would inform me that I need to roll it back?

[johnbelamaric]

I'm not sure why you're only talking about pod creation. What about service creation? Are there metrics there that could be useful? Do we have metrics around IP allocations? What about endpoint selection/creation? Do we have metrics about endpoints created?

[johnbelamaric]

It's not a service itself, but it can fail, right? You can fail to allocate the IP, you can fail to assign the IP to a Pod, you can fail to wire up the Pod sandbox, etc. These are likely already covered by existing metrics. Can you list them here, to inform operators what metrics they should be looking at? If they all of a sudden see a bunch of failures then they may want to turn it off.

What about watching for workload failures as I mentioned in my last review?

The point here is just to provide guidance and documentation. If you upgraded your cluster with 10,000 workloads on it, what would you look for to see that everything is going OK?

[johnbelamaric]

without my questions this is hard to follow, it needs more context. Even with my questions I am still not clear on how the answer addresses the "imbalance during re-enablement" case.

If that case is really possible (and I haven't seen anything to say whether or not it is), we just need to document it so people know what to expect and what to look for.

Once people upgrade, this goes on by default. If something goes wrong, and they turn it off, and then later want to re-enable it, we need to make sure they can plan appropriately. They may need to find any dual-stack services with bazillions of endpoints and make them single-stack first, for example. Just to avoid that imbalance situation.

I think the statement "we don't load balance across families" is trying to say that the imbalance is not possible. But I don't see how that invalidates the scenario I described. In fact, if we did load balance across families and did do protocol translation, then the scenario I described would not be a problem.

[johnbelamaric]

I don't think this is true. If the SLIs above are existing SLIs, you might just be able to say something like: existing kubelet pod creation or service creations SLOs (and whatever others may apply).

[johnbelamaric]

So, does CNI have any input from the kubelet as to whether Pods should be single or dual stack? Or is that completely out-of-band configuration that doesn't come from this feature at all? How does CNI know today which family of address to allocate?

[johnbelamaric]

on apiserver or on kubelet? Shouldn't it be everywhere? Otherwise I imagine really bad stuff can happen.

[johnbelamaric]

this question isn't about metrics server, it's about calls to the cloud provider. I believe in some cases IP allocation is done by the cloud provider? In GKE we have native IPs assigned to Pods. I don't know where that allocation is done exactly, maybe @bowei knows. But it seems feasible that dual stack would require an extra call to the cloud provider to allocation one of those native IPs.

[johnbelamaric]

I think this doesn't address workload failures nor the "imbalance", we probably want to discuss those.

[johnbelamaric]

What about workload failures? If I upgrade and suddently 10% of my workloads start crashlooping because they are stupid and don't know what to do with a AAAA record from DNS, I want to disable and/or rollback. Again this probably doesn't apply much in the upgrade case because there are no dual-stack services, but in the re-enablement case it does.

[aojea]

if you change the IPFamilyPolicy to SingleStack you will be back to the initial situation

[johnbelamaric]

Looking really good. A couple minor points.

[johnbelamaric]

A little confusing:

Pods and Services will remain single-stack until cli flags have been modified
as described in this KEP.

and

Pods will become dual-stack once CNI is configured for dual-stack.

are contradictory. I think the distinction is api vs pod runtime environment. Please clear that up.

[bridgetkromhout]

Thanks - edited.

[johnbelamaric]

Here is where you should mention:

DNS will immediately begin returning the secondary IP family, while endpoints, endpointSlices, and iptables programming may take some time. This can lead to large or very busy services from receiving excessive traffic on the secondary family address, until the endpoints, endpointslices, and ip tables rules are fully populated.

[johnbelamaric]

I wish we had a decent mitigation for this. It can happen anytime a larger service goes from single -> dual stack.

[bridgetkromhout]

Thanks - edited.

[johnbelamaric]

thank you for adding these

[johnbelamaric]

+1, nice

[johnbelamaric]

Thank you, looking good!

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dcbw, johnbelamaric, lachie83

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• keps/prod-readiness/OWNERS [johnbelamaric]
• keps/sig-network/OWNERS [dcbw,johnbelamaric]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[johnbelamaric]

And

/lgtm