Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make kube-proxy default to iptables (regression) #20464

Merged
merged 1 commit into from
Feb 3, 2016
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 0 additions & 1 deletion cmd/kube-proxy/app/options/options.go
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,6 @@ func NewProxyConfig() *ProxyServerConfig {
ResourceContainer: "/kube-proxy",
IPTablesSyncPeriod: unversioned.Duration{30 * time.Second},
UDPIdleTimeout: unversioned.Duration{250 * time.Millisecond},
Mode: componentconfig.ProxyModeUserspace,
ConntrackMax: 256 * 1024, // 4x default (64k)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you set explicitly to componentconfig.ProxyModeIPTables? That will show the correct defaulting in the --help output

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the reason to leave it unset is to allow that to mean "use what's best", which in this case translates to iptables. Also iptables auto downgrades to userspace if the wrong version of iptables is present. I agree, we need a clearer way to surface this. And a better way to test it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, it's probably worth adding a BestAvailable ProxyMode here. The doc is already wrong.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is wrong in the doc?

On Mon, Feb 1, 2016 at 10:21 PM, Mike Danese notifications@github.com
wrote:

In cmd/kube-proxy/app/options/options.go
#20464 (comment)
:

@@ -57,7 +57,6 @@ func NewProxyConfig() *ProxyServerConfig {
ResourceContainer: "/kube-proxy",
IPTablesSyncPeriod: unversioned.Duration{30 * time.Second},
UDPIdleTimeout: unversioned.Duration{250 * time.Millisecond},

  •       Mode:                           componentconfig.ProxyModeUserspace,
    

Ok, it's probably worth adding a BestAvailable ProxyMode here
https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/componentconfig/types.go#L65.
The doc is already wrong.


Reply to this email directly or view it on GitHub
https://github.com/kubernetes/kubernetes/pull/20464/files#r51528627.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahh, yes, will fix.

ConntrackTCPEstablishedTimeout: unversioned.Duration{Duration: 24 * time.Hour}, // 1 day (1/5 default)
},
Expand Down
11 changes: 9 additions & 2 deletions cmd/kube-proxy/app/server.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ package app

import (
"errors"
"fmt"
"net"
"net/http"
_ "net/http/pprof"
Expand Down Expand Up @@ -58,6 +59,7 @@ type ProxyServer struct {
Broadcaster record.EventBroadcaster
Recorder record.EventRecorder
Conntracker Conntracker // if nil, ignored
ProxyMode string
}

const (
Expand All @@ -83,6 +85,7 @@ func NewProxyServer(
broadcaster record.EventBroadcaster,
recorder record.EventRecorder,
conntracker Conntracker,
proxyMode string,
) (*ProxyServer, error) {
return &ProxyServer{
Client: client,
Expand All @@ -92,6 +95,7 @@ func NewProxyServer(
Broadcaster: broadcaster,
Recorder: recorder,
Conntracker: conntracker,
ProxyMode: proxyMode,
}, nil
}

Expand Down Expand Up @@ -248,7 +252,7 @@ func NewProxyServerDefault(config *options.ProxyServerConfig) (*ProxyServer, err

conntracker := realConntracker{}

return NewProxyServer(client, config, iptInterface, proxier, eventBroadcaster, recorder, conntracker)
return NewProxyServer(client, config, iptInterface, proxier, eventBroadcaster, recorder, conntracker, proxyMode)
}

// Run runs the specified ProxyServer. This should never exit (unless CleanupAndExit is set).
Expand All @@ -265,8 +269,11 @@ func (s *ProxyServer) Run() error {

s.Broadcaster.StartRecordingToSink(s.Client.Events(""))

// Start up Healthz service if requested
// Start up a webserver if requested
if s.Config.HealthzPort > 0 {
http.HandleFunc("/proxyMode", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "%s", s.ProxyMode)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we actually store what getProxyMode returns back into this? (I haven't checked)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More directly, this will say userspace if we don't have the right iptables version right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The NewProxyServer() call sets it.

})
go util.Until(func() {
err := http.ListenAndServe(s.Config.HealthzBindAddress+":"+strconv.Itoa(s.Config.HealthzPort), nil)
if err != nil {
Expand Down
4 changes: 2 additions & 2 deletions docs/admin/kube-proxy.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,12 +71,12 @@ kube-proxy
--masquerade-all[=false]: If using the pure iptables proxy, SNAT everything
--master="": The address of the Kubernetes API server (overrides any value in kubeconfig)
--oom-score-adj=-999: The oom-score-adj value for kube-proxy process. Values must be within the range [-1000, 1000]
--proxy-mode=userspace: Which proxy mode to use: 'userspace' (older) or 'iptables' (faster). If blank, look at the Node object on the Kubernetes API and respect the 'net.experimental.kubernetes.io/proxy-mode' annotation if provided. Otherwise use the best-available proxy (currently iptables). If the iptables proxy is selected, regardless of how, but the system's kernel or iptables versions are insufficient, this always falls back to the userspace proxy.
--proxy-mode=: Which proxy mode to use: 'userspace' (older) or 'iptables' (faster). If blank, look at the Node object on the Kubernetes API and respect the 'net.experimental.kubernetes.io/proxy-mode' annotation if provided. Otherwise use the best-available proxy (currently iptables). If the iptables proxy is selected, regardless of how, but the system's kernel or iptables versions are insufficient, this always falls back to the userspace proxy.
--proxy-port-range=: Range of host ports (beginPort-endPort, inclusive) that may be consumed in order to proxy service traffic. If unspecified (0-0) then ports will be randomly chosen.
--udp-timeout=250ms: How long an idle UDP connection will be kept open (e.g. '250ms', '2s'). Must be greater than 0. Only applicable for proxy-mode=userspace
```

###### Auto generated by spf13/cobra on 27-Jan-2016
###### Auto generated by spf13/cobra on 1-Feb-2016


<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
Expand Down
4 changes: 2 additions & 2 deletions pkg/apis/componentconfig/types.go
Original file line number Diff line number Diff line change
Expand Up @@ -63,9 +63,9 @@ type KubeProxyConfiguration struct {
}

// Currently two modes of proxying are available: 'userspace' (older, stable) or 'iptables'
// (experimental). If blank, look at the Node object on the Kubernetes API and respect the
// (newer, faster). If blank, look at the Node object on the Kubernetes API and respect the
// 'net.experimental.kubernetes.io/proxy-mode' annotation if provided. Otherwise use the
// best-available proxy (currently userspace, but may change in future versions). If the
// best-available proxy (currently iptables, but may change in future versions). If the
// iptables proxy is selected, regardless of how, but the system's kernel or iptables
// versions are insufficient, this always falls back to the userspace proxy.
type ProxyMode string
Expand Down
2 changes: 1 addition & 1 deletion pkg/kubemark/hollow_proxy.go
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ func NewHollowProxyOrDie(
endpointsConfig.Channel("api"),
)

hollowProxy, err := proxyapp.NewProxyServer(client, config, iptInterface, &FakeProxier{}, broadcaster, recorder, nil)
hollowProxy, err := proxyapp.NewProxyServer(client, config, iptInterface, &FakeProxier{}, broadcaster, recorder, nil, "fake")
if err != nil {
glog.Fatalf("Error while creating ProxyServer: %v\n", err)
}
Expand Down
11 changes: 11 additions & 0 deletions test/e2e/kubeproxy.go
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,10 @@ func (config *KubeProxyTestConfig) hitNodePort(epCount int) {
config.dialFromNode("udp", node2_IP, nodeUdpPort, tries, epCount)
By("dialing(http) node1 --> node2:nodeHttpPort")
config.dialFromNode("http", node2_IP, nodeHttpPort, tries, epCount)

By("checking kube-proxy URLs")
config.getSelfURL("/healthz", "ok")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make this readiness?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To what end? The kube-proxy isn't behind a Service, so not being ready won't actually prevent anything from happening or enable anything, will it?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can poll for ready though, from the test. I think the reboot tests do this and if you don't have a probe the kubelet just starts you off as ready.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that this test does NO retries or readiness, I'd like to defer that
extra work and get this and other PRs in before freeze.

On Tue, Feb 2, 2016 at 1:36 PM, Prashanth B notifications@github.com
wrote:

In test/e2e/kubeproxy.go
#20464 (comment)
:

@@ -197,6 +197,10 @@ func (config *KubeProxyTestConfig) hitNodePort(epCount int) {
config.dialFromNode("udp", node2_IP, nodeUdpPort, tries, epCount)
By("dialing(http) node1 --> node2:nodeHttpPort")
config.dialFromNode("http", node2_IP, nodeHttpPort, tries, epCount)
+

  • By("checking kube-proxy URLs")
  • config.getSelfURL("/healthz", "ok")

you can poll for ready though, from the test. I think the reboot tests
do this and if you don't have a probe the kubelet just starts you off as
ready.


Reply to this email directly or view it on GitHub
https://github.com/kubernetes/kubernetes/pull/20464/files#r51639186.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with the defer. The reboot test will poll till all pods are ready before declaring success.

config.getSelfURL("/proxyMode", "iptables") // the default
}

func (config *KubeProxyTestConfig) hitEndpoints() {
Expand Down Expand Up @@ -252,6 +256,13 @@ func (config *KubeProxyTestConfig) dialFromNode(protocol, targetIP string, targe
Expect(strconv.Atoi(strings.TrimSpace(stdout))).To(BeNumerically("==", expectedCount))
}

func (config *KubeProxyTestConfig) getSelfURL(path string, expected string) {
cmd := fmt.Sprintf("curl -s --connect-timeout 1 http://localhost:10249%s", path)
By(fmt.Sprintf("Getting kube-proxy self URL %s", path))
stdout := RunHostCmdOrDie(config.f.Namespace.Name, config.hostTestContainerPod.Name, cmd)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this doesn't retry and I don't think our kube-proxy has a readiness probe. So there's a chance the server isn't up if we run after a reboot test?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

None of this test retries. Different issue.

Expect(strings.Contains(stdout, expected)).To(BeTrue())
}

func (config *KubeProxyTestConfig) createNetShellPodSpec(podName string, node string) *api.Pod {
pod := &api.Pod{
TypeMeta: unversioned.TypeMeta{
Expand Down