No timeout when Kubelet Calling cni plugin

[liucimin]

BUG REPORT:

/kind bug

What happened:
When the kubelet create sandbox by cni plugin.
It will just use exec.Cmd to call the cni binary file.
In some cni plugin,such as contiv,the cni binary file may be locked and no return.
In this case, when kubelet first time call RunPodSandbox got a ctx timeout. the next time kubelet re-creates PodSandbox, it will always report this timeout.

What you expected to happen:
Next time kubelet re-creates PodSandbox it will do call cni again.

How to reproduce it (as minimally and precisely as possible):
Create a pod in a node,and make the cni binary file not return.

Environment:

• Kubernetes version (use kubectl version):v.1.9.2

@kubernetes/sig-node-bugs

[liucimin]

/assign liucimin

[liucimin]

/sig node

[huydinhle]

we are also got affected by this issue #45419, can someone take a look at this PR. we will be happy to test this out

[liucimin]

@huydinhle
Yes, i m trying to fix this problem.
My change is first set a timeout in cni.
When the cni merge my code,then kubelet will fix more easy.
containernetworking/cni#568

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[nikopen]

containernetworking/cni#568 got merged, this is resolved I presume? @liucimin

/close

[k8s-ci-robot]

@nikopen: Closing this issue.

In response to this:

containernetworking/cni#568 got merged, this is resolved I presume? @liucimin

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[nikopen]

Just saw that a fix is needed in kubelet as well, can you confirm @liucimin ?

/reopen

[k8s-ci-robot]

@nikopen: Reopening this issue.

In response to this:

Just saw that a fix is needed in kubelet as well, can you confirm @liucimin ?

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[bboreham]

Yes kubelet needs to be changed.
And add a test for this case.

[liucimin]

@nikopen
Yes,kubelet should update the cni version.
I will change the kubelet when the cni project release a new version

[nikopen]

@bboreham what are the plans for cutting a new CNI release and bumping it in k8s? Does it need a new major k8s release to be bumped, v1.13?

Here I can see the latest is v0.7.3, seen as plain 'cni' in fedora, but in the official page the latest is 0.6.0 with some 0.7.0-alpha versions. Maybe because on the v1.12 changelogs I can see k8s is still using 0.6.0, but it's still inconsistent?

It's an old issue affecting many ( #45419 ), if the fix in kubelet is straightforward then this can be a great addition, given v1.13 is marked as a stability release.

[nikopen]

/remove-lifecycle stale

[bboreham]

Please can you check that all necessary changes are made in Kubernetes. The CNI change is all in the library so just vendor in whatever commit you want to, and check it all works as you need.

Here I can see the latest is v0.7.3

That's the plugins - different thing.

[nikopen]

@liucimin feel free to prepare the Kubelet PR, it will then be easier to coordinate the full fix.

[liucimin]

@nikopen
yes, i just waiting for the cni plugin release a new version.Then i will upgrade our kubelet's dockeshim to
use the new cni plugin.

[bboreham]

There is no relevant change in CNI plugins.

The change you are waiting for is in libcni, and is already merged to master. Do not wait for it: test at will.

(I expect the calling code will need changing to make use of the new timeout, but I haven’t studied it)

[roywangtj]

The .go files that are under [https://github.com/kubernetes/kubernetes/tree/master/vendor/github.com/containernetworking/cni/libcni] were all two years ago. Where can we find the libcni code merged?

[bboreham]

@roywangti the repo for libcni is https://github.com/containernetworking/cni/

Part of the next step is to copy updated files into the location you pointed at.

[roywangtj]

@bboreham, got it, Thanks!

[liucimin]

#71653

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[ijumps]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[liucimin]

/reopen
/remove-lifecycle rotten

[k8s-ci-robot]

@liucimin: Reopened this issue.

In response to this:

/reopen
/remove-lifecycle rotten

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.