New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conntrack work-around no longer works #50

Open
pmichali opened this Issue Nov 28, 2017 · 2 comments

Comments

Projects
None yet
1 participant
@pmichali
Contributor

pmichali commented Nov 28, 2017

In wrapkubeadm, the script passes --conntrack-max=0 and --conntrack-max-per-core=0 to kube-proxy in an attempt to tell it to skip trying to update the hashsize, when there is a large conntrack max configured on the host to avoid docker issue moby/moby#24000.

Unfortunately, recent changes to kube-proxy have it read the CLI args, and then attempt to read from a config file. If there is a config file (there always is), the CLI args are ignored. As a result, with the reading of the config file, there are no conntrack settings so the default conntrack max-per-core value of 32768 is used. When on a system with 32 CPUs, the resulting conntrack value (1048576) can be more than four times larger than the hashsize (e.g. 262144), which causes kube-proxy to attempt to increase the hashsize and hits the docker issue.

DinD needs to be able to set both conntrack max and max-per-core to zero in the config file, which tells kube-proxy to ignore attempting to modify the hashsize in the condition where there is a large number of conntracks.

@pmichali

This comment has been minimized.

Contributor

pmichali commented Nov 28, 2017

As a temporary work-around, one can change kube-proxy to force the default conntrack settings to zero, which will prevent kube-proxy from trying to update hashsize. The BUILD_HYPERKUBE and BUILD_KUBEADM flags can be set and the following patch applied to a local kubernetes repo:

--- a/cmd/kubeadm/app/apis/kubeadm/v1alpha1/defaults.go
+++ b/cmd/kubeadm/app/apis/kubeadm/v1alpha1/defaults.go
@@ -133,6 +133,14 @@ func SetDefaults_ProxyConfiguration(obj *MasterConfiguration) {
                obj.KubeProxy.Config.ClientConnection.KubeConfigFile = KubeproxyKubeConfigFileName
        }

+       if obj.KubeProxy.Config.Conntrack.Max == nil {
+           obj.KubeProxy.Config.Conntrack.Max = utilpointer.Int32Ptr(0)
+       }
+
+       if obj.KubeProxy.Config.Conntrack.MaxPerCore == nil {
+           obj.KubeProxy.Config.Conntrack.MaxPerCore = utilpointer.Int32Ptr(0)
+       }
+
        kubeproxyscheme.Scheme.Default(obj.KubeProxy.Config)
 }
@pmichali

This comment has been minimized.

Contributor

pmichali commented Nov 28, 2017

Just to clarify on the docker issue. The current code, does a check to see if the filesystem is writeable, in an attempt to detect when there is a nested docker environment and to be able to avoid the docker issue. However, what is happening, is that the filesystem is reporting as writeable, but later, when the writing is attempted, it fails. So, the code in kube-proxy to address the docker issue, does not work.

There was an issue about the file system not allowing writing (kubernetes/kubernetes#25543 (comment)). There was no solution to that issue, so the work around is to set conntrack max and max-per-core to zero to avoid changing the hashsize. This could be done in a nested docker situation (like DinD), and was, via the CLI arguments, but that solution will no longer work. Instead, the workaround must be employed via the config file, which is the request of this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment