Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Init Container will be removed by kubeGC when we set --maximum-dead-containers-per-container=0 #77859

Open
Pingan2017 opened this issue May 14, 2019 · 9 comments

Comments

Projects
None yet
3 participants
@Pingan2017
Copy link
Member

commented May 14, 2019

What happened:
1.set --maximum-dead-containers-per-container=0
2. create a deploy with a init container
3. watch the pod status , you will see the change

[root@hpa-vm:~]$ kubectl get pod -w
NAME          READY   STATUS    RESTARTS   AGE
nginx-f8jfv   1/1     Running   0          30s
NAME          READY   STATUS     RESTARTS   AGE
nginx-f8jfv   0/1     Init:0/1   0          95s
nginx-f8jfv   1/1     Running    0          100s

What you expected to happen:
init container wont't be removed.
we need separate the containerGC and initContainerGC

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):
  • Cloud provider or hardware configuration:
  • OS (e.g: cat /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Network plugin and version (if this is a network-related bug):
  • Others:
@Pingan2017

This comment has been minimized.

Copy link
Member Author

commented May 14, 2019

@Pingan2017

This comment has been minimized.

Copy link
Member Author

commented May 14, 2019

@mattjmcnaughton

This comment has been minimized.

Copy link
Contributor

commented May 14, 2019

Hi @Pingan2017 :)

Thanks for sharing this.

Do you mind providing a little more detail on the actual behavior and the expected behavior? I think I'm not totally following why its a problem to remove the init container if its already completed successfully. Is the issue that Kubelet thinks it hasn't completed successfully and keeps on trying to rerun it, leading to repeated execution of the init container?

@Pingan2017

This comment has been minimized.

Copy link
Member Author

commented May 15, 2019

Is the issue that Kubelet thinks it hasn't completed successfully and keeps on trying to rerun it, leading to repeated execution of the init container?

Yes! Kubelet will run the init container repeatedly.
pod flapping between Running/Init

[root@hpa-vm:~]$ kubectl get pod -w
NAME          READY   STATUS    RESTARTS   AGE
nginx-f8jfv   1/1     Running   0          17h
NAME          READY   STATUS     RESTARTS   AGE
nginx-f8jfv   0/1     Init:0/1   0          17h
nginx-f8jfv   1/1     Running    0          17h
nginx-f8jfv   0/1     Init:0/1   0          17h
nginx-f8jfv   1/1     Running    0          17h
@mattjmcnaughton

This comment has been minimized.

Copy link
Contributor

commented May 15, 2019

/assign

Thanks for helping me understand @Pingan2017 !

It seems like there are two possible solutions here:

  1. As you suggested, we could separate init container clean up, and either not apply maximum dead containers per container or use max(maximum_dead_containers_per_container, 1) for init containers.
  2. Use a different method than dead container existence for determining whether an init container has been run. This seems more complex, although ultimately more robust.

Thoughts? I'm leaning towards option 1 (and am happy to take a shot at implementing it), but would love to hear from others.

@mattjmcnaughton

This comment has been minimized.

Copy link
Contributor

commented May 20, 2019

I was doing a little more research on this, and I realized that --maximum-dead-containers-per-container has been marked deprecated, with the recommendation to instead use --eviction-hard or --eviction-soft.

@Pingan2017 do you know if this problem remains when using --eviction-hard/--eviction-soft? If no, thoughts on closing this issue in favor recommending anyone experiencing this issue utilize the non-deprecated behavior?

Here's some more info on --eviction-hard and --eviction-soft (https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/#eviction-policy).

@Pingan2017

This comment has been minimized.

Copy link
Member Author

commented May 22, 2019

@mattjmcnaughton
I'm afraid not. Eviction also call the kubeGC to delete containers before evict pods

@Pingan2017

This comment has been minimized.

Copy link
Member Author

commented May 22, 2019

@kubernetes/sig-node-bugs Could someone take a look?

@mattjmcnaughton

This comment has been minimized.

Copy link
Contributor

commented May 22, 2019

@mattjmcnaughton
I'm afraid not. Eviction also call the kubeGC to delete containers before evict pods

Hmm, but if the kubeGC evicts the pods, than wouldn't we expect it to delete the init container?

At a higher level, can you share the use case for setting --maximum-dead-containers-per-container=0, and confirm that we couldn't accomplish something similar by not setting --maximum-dead-containers-per-container=0 and instead setting --eviction-hard.

Definitely not opposed to looking into this further, but since the functionality is deprecated, just want to confirm its a sensible investment. Thanks for your patience :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.