Wow, this is a huge increase.
@squeed Any comments on the performances? I wonder whether this would have any impacts on kubelet and GC.
/cc @luxas

Interesting, taking a look.

@squeed Thanks for your help.

Ah, I think I know the issue. The new CNI plugins wait for DAD to finish before proceeding, which takes 3 seconds on Linux. Still need to do some testing (helpfully, my laptop drive died yesterday).

The address-settle adds about 1 second of delay. Is that likely the complete cause, or is it also something else? (I'm not familiar with this test)

Ah, network setup is also serialized behind a mutex; the increase from 20ms to 1020ms for network setup in the critical path is almost certainly the culprit.

The CNI plugins could avoid the DAD delay for now (since we're not doing IPv6), this will eventually become a problem when ipv6 is supported.

Since the CNI plugin execution itself doesn't need to be synchronized, I think I can move that outside the mutex.

This seems to be a v1.9-blocking issue that needs to be fixed/tracked, right?

@dnardo @freehan

Good catch... This is why we need to bump CNI version at the early stage of a release cycle.

@squeed sig-network is planning to add ipv6 alpha support for the 1.9 release.

/reopen

@porridge: you can't re-open an issue/PR unless you authored it or you are assigned to it.

/open

To be honest it does not seem like #54800 has fixed this. We are still seeing high pod startup in large ane medium scale tests: #55060 (comment)

Crazy idea: Can we do things conditionally for IPv4 and IPv6?
Does this only affect kubenet or any CNI plugin (like Weave, Flannel, etc)?

We disable DAD when using kubenet. Since Kubenet has external guarantees that addresses won't collide, this is safe.

I thought kubenet would be deprecated sooner or later. If that's still true, does that mean we'd need yet another solution for a non-kubenet setup?

Another question from a layman: do we know why it's so slow on Linux? Is it talking to other nodes (something like gratitious ARP)? Does it have a chance to become faster in the future? Does the delay depend on the size of the network/number of nodes, etc?

One way to make it faster on Linux is to enable "optimistic DAD" (RFC4429), which allows usage of tentative addresses. This is still considered to be an "experimental" feature, and is not enabled in all kernels. IIRC, the latest Ubuntu still has it disabled. CoreOS CL, Debian do have it. This is a sysctl that would need to be set in the container's network namespace, since it's a namespaced sysctl (and new network namespaces take the default configuration, rather than inheriting from the host).

@yujuhong yeah, you're right. Hopefully by then optimistic_dad is enabled on all kernels...

I thought kubenet would be deprecated sooner or later.

it will

If that's still true

yes

does that mean we'd need yet another solution for a non-kubenet setup

probably, like @squeed said above.

pinging @kubernetes/kubernetes-release-managers already -- this MIGHT be one of the big things we need to notify people about if this doesn't get solved.

[MILESTONENOTIFIER] Milestone Issue Current

@dashpole

@spiffxp Could you approve this for the milestone?

This issue needs to be resolved urgently, as it's causing 5k-node performance job (which is release-blocking) to fail.

I'm working on disabling dad in kubenet (option 3).

Should be ready soon.

OK, PR filed.

/status approved-for-milestone
FYI @shyamjvs it looks like @luxas already added to this to the v1.9 milestone, and @dchen1107 already added the label manually.

PR #55247 seems to have done the trick. The latency has decreased

from:

    {
      "data": {
        "Perc100": 4762.418815,
        "Perc50": 3308.810795,
        "Perc90": 3955.747742,
        "Perc99": 4756.553711
      },

to:

    {
      "data": {
        "Perc100": 3178.085005,
        "Perc50": 2186.109036,
        "Perc90": 2768.180504,
        "Perc99": 3126.759942
      },

Closing the issue. Thanks everyone for working on this!

@shyamjvs Do we have any coverage what happens with these numbers if you're using a 3rd party CNI provider (Weave, Calico, Flannel, whatever)? If this ONLY fixes kubenet but throws a perf penalty to everyone non-GCE, I think we should reopen this issue.

The delay was on the CNI executable side; so it depends on the plugin being used. That said, it would be good to have that codepath under test in general.

We're not scale-testing with other CNI providers. One, because it seems to be the default option being used at head. Two, because it's simply not possible to scale test different configurations exhaustively due to resource and time constraints of running large cluster tests. So we choose to validate against (mostly) the default configs.

That said, opening this issue to study those options sg. However, it would be hard to get this under sig-scale's purview atm.

Ok, thanks.
cc @timothysc FYI

I'd suggest opening a separate issue to track this, as it's unlikely to fit
in 1.9.

+1. Is this tracked anywhere?