Endpoints are missed when creating many services fast

[uablrek]

I create many pods and services in one command by apply a directory;

kubectl apply -f /etc/kubernetes/k8s-test/

Most services has 4 endpoints in my setup but when accessing only a few is used;

vm-003 ~ # mconnect -address mserver-deployment.default.svc.xcluster:5001 -nconn 100
Failed connects; 0
Failed reads; 0
mserver-deployment-654bbdc6dc-xj2fk 100
vm-003 ~ # mconnect -address mserver.default.svc.xcluster:5001 -nconn 100
Failed connects; 0
Failed reads; 0
mserver-mmd8f 52
mserver-sjrls 48

Both services has 4 endpoints. I waited for ~30 min to see if sync would fix this but the problem remains.

Output from print-state: print-state.txt

[uablrek]

Copied from the old repo

[uablrek]

Restarting kpng fixes the problem.

[mcluseau]

The output from print-state shows 4 endpoints per service so it seems the issue is either in full state sink or the nft backend.

…

On Fri, Apr 23, 2021, 19:10 Lars Ekman ***@***.***> wrote: Restarting kpng fixes the problem. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA5QROAFLB2XDHKSNSDI6VDTKGSXJANCNFSM43O4X6XQ> .

[mcluseau]

I've found a bug that has a really high probability to have caused your issue (not a race condition though). It's worth trying again :)

[uablrek]

No, still the same problem. Re-built on;

commit 95df644fd8750210c91bd6c938ca74ce768cbac0 (HEAD -> master, origin/master, origin/HEAD)
Author: Mikaël Cluseau <mikael.cluseau@gmail.com>
Date:   Fri May 7 22:54:44 2021 +0200

    api2store: fix tls config build

Is there a kpng version command of some kind, showing version or git commit? If not, it should be IMHO.

[uablrek]

I am testing dual-stack, with ipFamilies and ipFamilyPolicy here.
Is kpng and the nft backend prepared for that? Or am I tesing a Not-Implemented-Yet feature?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[mcluseau]

I think we need the test yaml definitions to move forward here.

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.