[WIP] Nftables proxy #124272

aojea · 2024-04-11T14:04:16Z

/kind feature

NONE

k8s-ci-robot · 2024-04-11T14:04:20Z

Please note that we're already in Test Freeze for the release-1.30 branch. This means every merged PR will be automatically fast-forwarded via the periodic ci-fast-forward job to the release branch of the upcoming v1.30.0 release.

Fast forwards are scheduled to happen every 6 hours, whereas the most recent run was: Thu Apr 11 13:45:22 UTC 2024.

k8s-ci-robot · 2024-04-11T14:04:25Z

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

aojea · 2024-04-11T14:04:44Z

/hold
/cc @danwinship

let's start to get some data

aojea · 2024-04-11T14:04:55Z

/test

k8s-ci-robot · 2024-04-11T14:04:58Z

@aojea: The /test command needs one or more targets.
The following commands are available to trigger required jobs:

/test pull-cadvisor-e2e-kubernetes
/test pull-cos-containerd-e2e-ubuntu-gce
/test pull-kubernetes-conformance-kind-ga-only-parallel
/test pull-kubernetes-coverage-unit
/test pull-kubernetes-dependencies
/test pull-kubernetes-dependencies-go-canary
/test pull-kubernetes-e2e-gce
/test pull-kubernetes-e2e-gce-100-performance
/test pull-kubernetes-e2e-gce-big-performance
/test pull-kubernetes-e2e-gce-cos
/test pull-kubernetes-e2e-gce-cos-canary
/test pull-kubernetes-e2e-gce-cos-no-stage
/test pull-kubernetes-e2e-gce-network-proxy-http-connect
/test pull-kubernetes-e2e-gce-scale-performance-manual
/test pull-kubernetes-e2e-kind
/test pull-kubernetes-e2e-kind-ipv6
/test pull-kubernetes-integration
/test pull-kubernetes-integration-go-canary
/test pull-kubernetes-kubemark-e2e-gce-scale
/test pull-kubernetes-node-e2e-containerd
/test pull-kubernetes-typecheck
/test pull-kubernetes-unit
/test pull-kubernetes-unit-go-canary
/test pull-kubernetes-update
/test pull-kubernetes-verify
/test pull-kubernetes-verify-go-canary
/test pull-kubernetes-verify-lint

The following commands are available to trigger optional jobs:

/test check-dependency-stats
/test pull-ci-kubernetes-unit-windows
/test pull-crio-cgroupv1-node-e2e-eviction
/test pull-crio-cgroupv1-node-e2e-features
/test pull-crio-cgroupv1-node-e2e-hugepages
/test pull-crio-cgroupv1-node-e2e-resource-managers
/test pull-crio-cgroupv2-imagefs-separatedisktest
/test pull-crio-cgroupv2-node-e2e-eviction
/test pull-crio-cgroupv2-splitfs-splitdisk
/test pull-e2e-gce-cloud-provider-disabled
/test pull-kubernetes-conformance-image-test
/test pull-kubernetes-conformance-kind-ga-only
/test pull-kubernetes-conformance-kind-ipv6-parallel
/test pull-kubernetes-cos-cgroupv1-containerd-node-e2e
/test pull-kubernetes-cos-cgroupv1-containerd-node-e2e-features
/test pull-kubernetes-cos-cgroupv2-containerd-node-e2e
/test pull-kubernetes-cos-cgroupv2-containerd-node-e2e-eviction
/test pull-kubernetes-cos-cgroupv2-containerd-node-e2e-features
/test pull-kubernetes-cos-cgroupv2-containerd-node-e2e-serial
/test pull-kubernetes-crio-node-memoryqos-cgrpv2
/test pull-kubernetes-cross
/test pull-kubernetes-e2e-autoscaling-hpa-cm
/test pull-kubernetes-e2e-autoscaling-hpa-cpu
/test pull-kubernetes-e2e-autoscaling-vpa-full
/test pull-kubernetes-e2e-capz-azure-disk
/test pull-kubernetes-e2e-capz-azure-disk-vmss
/test pull-kubernetes-e2e-capz-azure-file
/test pull-kubernetes-e2e-capz-azure-file-vmss
/test pull-kubernetes-e2e-capz-conformance
/test pull-kubernetes-e2e-capz-windows-alpha-feature-vpa
/test pull-kubernetes-e2e-capz-windows-alpha-features
/test pull-kubernetes-e2e-capz-windows-master
/test pull-kubernetes-e2e-capz-windows-serial-slow
/test pull-kubernetes-e2e-capz-windows-serial-slow-hpa
/test pull-kubernetes-e2e-containerd-gce
/test pull-kubernetes-e2e-ec2
/test pull-kubernetes-e2e-ec2-arm64
/test pull-kubernetes-e2e-ec2-conformance
/test pull-kubernetes-e2e-ec2-conformance-arm64
/test pull-kubernetes-e2e-ec2-device-plugin-gpu
/test pull-kubernetes-e2e-gce-canary
/test pull-kubernetes-e2e-gce-correctness
/test pull-kubernetes-e2e-gce-cos-alpha-features
/test pull-kubernetes-e2e-gce-csi-serial
/test pull-kubernetes-e2e-gce-device-plugin-gpu
/test pull-kubernetes-e2e-gce-disruptive-canary
/test pull-kubernetes-e2e-gce-kubelet-credential-provider
/test pull-kubernetes-e2e-gce-network-policies
/test pull-kubernetes-e2e-gce-network-proxy-grpc
/test pull-kubernetes-e2e-gce-providerless
/test pull-kubernetes-e2e-gce-serial
/test pull-kubernetes-e2e-gce-serial-canary
/test pull-kubernetes-e2e-gce-storage-disruptive
/test pull-kubernetes-e2e-gce-storage-slow
/test pull-kubernetes-e2e-gce-storage-snapshot
/test pull-kubernetes-e2e-gci-gce-autoscaling
/test pull-kubernetes-e2e-gci-gce-ingress
/test pull-kubernetes-e2e-gci-gce-ipvs
/test pull-kubernetes-e2e-inplace-pod-resize-containerd-main-v2
/test pull-kubernetes-e2e-kind-alpha-beta-features
/test pull-kubernetes-e2e-kind-alpha-features
/test pull-kubernetes-e2e-kind-beta-features
/test pull-kubernetes-e2e-kind-canary
/test pull-kubernetes-e2e-kind-cloud-provider-loadbalancer
/test pull-kubernetes-e2e-kind-dual-canary
/test pull-kubernetes-e2e-kind-evented-pleg
/test pull-kubernetes-e2e-kind-ipv6-canary
/test pull-kubernetes-e2e-kind-ipvs-dual-canary
/test pull-kubernetes-e2e-kind-kms
/test pull-kubernetes-e2e-kind-multizone
/test pull-kubernetes-e2e-kind-nftables
/test pull-kubernetes-e2e-storage-kind-alpha-features
/test pull-kubernetes-e2e-storage-kind-disruptive
/test pull-kubernetes-kind-dra
/test pull-kubernetes-kind-json-logging
/test pull-kubernetes-kind-text-logging
/test pull-kubernetes-kubemark-e2e-gce-big
/test pull-kubernetes-linter-hints
/test pull-kubernetes-local-e2e
/test pull-kubernetes-node-arm64-e2e-containerd-ec2
/test pull-kubernetes-node-arm64-e2e-containerd-serial-ec2
/test pull-kubernetes-node-arm64-ubuntu-serial-gce
/test pull-kubernetes-node-crio-cgrpv1-evented-pleg-e2e
/test pull-kubernetes-node-crio-cgrpv2-e2e
/test pull-kubernetes-node-crio-cgrpv2-e2e-kubetest2
/test pull-kubernetes-node-crio-cgrpv2-imagefs-e2e
/test pull-kubernetes-node-crio-cgrpv2-splitfs-e2e
/test pull-kubernetes-node-crio-e2e
/test pull-kubernetes-node-crio-e2e-kubetest2
/test pull-kubernetes-node-e2e-containerd-1-7-dra
/test pull-kubernetes-node-e2e-containerd-alpha-features
/test pull-kubernetes-node-e2e-containerd-ec2
/test pull-kubernetes-node-e2e-containerd-features
/test pull-kubernetes-node-e2e-containerd-features-kubetest2
/test pull-kubernetes-node-e2e-containerd-kubetest2
/test pull-kubernetes-node-e2e-containerd-serial-ec2
/test pull-kubernetes-node-e2e-containerd-standalone-mode
/test pull-kubernetes-node-e2e-containerd-standalone-mode-all-alpha
/test pull-kubernetes-node-e2e-crio-dra
/test pull-kubernetes-node-kubelet-containerd-flaky
/test pull-kubernetes-node-kubelet-credential-provider
/test pull-kubernetes-node-kubelet-serial-containerd
/test pull-kubernetes-node-kubelet-serial-containerd-alpha-features
/test pull-kubernetes-node-kubelet-serial-containerd-kubetest2
/test pull-kubernetes-node-kubelet-serial-containerd-sidecar-containers
/test pull-kubernetes-node-kubelet-serial-cpu-manager
/test pull-kubernetes-node-kubelet-serial-cpu-manager-kubetest2
/test pull-kubernetes-node-kubelet-serial-crio-cgroupv1
/test pull-kubernetes-node-kubelet-serial-crio-cgroupv2
/test pull-kubernetes-node-kubelet-serial-hugepages
/test pull-kubernetes-node-kubelet-serial-memory-manager
/test pull-kubernetes-node-kubelet-serial-pod-disruption-conditions
/test pull-kubernetes-node-kubelet-serial-topology-manager
/test pull-kubernetes-node-kubelet-serial-topology-manager-kubetest2
/test pull-kubernetes-node-swap-conformance-fedora-serial
/test pull-kubernetes-node-swap-conformance-ubuntu-serial
/test pull-kubernetes-node-swap-fedora
/test pull-kubernetes-node-swap-fedora-serial
/test pull-kubernetes-node-swap-ubuntu-serial
/test pull-kubernetes-unit-experimental
/test pull-publishing-bot-validate

Use /test all to run the following jobs that were automatically triggered:

pull-kubernetes-conformance-kind-ga-only-parallel
pull-kubernetes-dependencies
pull-kubernetes-e2e-ec2
pull-kubernetes-e2e-ec2-conformance
pull-kubernetes-e2e-gce
pull-kubernetes-e2e-gce-canary
pull-kubernetes-e2e-gce-cos-alpha-features
pull-kubernetes-e2e-kind
pull-kubernetes-e2e-kind-ipv6
pull-kubernetes-integration
pull-kubernetes-linter-hints
pull-kubernetes-node-e2e-containerd
pull-kubernetes-typecheck
pull-kubernetes-unit
pull-kubernetes-verify
pull-kubernetes-verify-lint

In response to this:

/test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2024-04-11T14:05:16Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: aojea
Once this PR has been reviewed and has the lgtm label, please assign mwielgus for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

aojea · 2024-04-11T14:06:19Z

/test pull-kubernetes-e2e-gce-100-performance

aojea · 2024-04-11T15:43:18Z

/test pull-kubernetes-e2e-gce-100-performance

aojea · 2024-04-11T17:56:38Z

https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/124272/pull-kubernetes-e2e-gce-100-performance/1778448759718416384

    {
      "data": {
        "Perc50": 1519.802112,
        "Perc90": 2282.213099,
        "Perc99": 6815.068537
      },
      "unit": "ms",
      "labels": {
        "Metric": "NetworkProgrammingLatency"
      }

This is much better, but iptables runs with 10s of sync period, last time we analyzed this the latency in iptables improved but CPU increased #110268 (comment)
#114229 (comment)
#115720

      "data": {
        "Perc50": 12.5,
        "Perc90": 22.5,
        "Perc99": 24.75
      },
      "unit": "ms",
      "labels": {
        "Metric": "InClusterNetworkLatency"
      }

is the same https://perf-dash.k8s.io/#/?jobname=gce-100Nodes-master&metriccategoryname=Network&metricname=Load_NetworkLatency&Metric=InClusterNetworkLatency

The sync rules in nftables

The sync rules in iptables from https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/1778462542213943296 :) that are way better

k8s-ci-robot · 2024-04-11T19:15:02Z

@aojea: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kubernetes-unit	`fa74ceb`	link	true	`/test pull-kubernetes-unit`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

danwinship · 2024-04-11T19:33:03Z

https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/124272/pull-kubernetes-e2e-gce-100-performance/1778448759718416384
    {
      "data": {
        "Perc50": 1519.802112,
        "Perc90": 2282.213099,
        "Perc99": 6815.068537
      },
      "unit": "ms",
      "labels": {
        "Metric": "NetworkProgrammingLatency"
      }

(So that's the NetworkProgrammingLatency numbers from the e2e-gce-100-performance run you kicked off in this PR, aka "length of time from when the EndpointSliceController makes a change to when kube-proxy programs the change in nftables".)

This is much better [than equivalent current iptables]

For comparision from a recent run:

    {
      "data": {
        "Perc50": 9034.440056,
        "Perc90": 10284.454043,
        "Perc99": 10984.283114
      },
      "unit": "ms",
      "labels": {
        "Metric": "NetworkProgrammingLatency"
      }
    }

but iptables runs with 10s of sync period

(which is to say, the job runs with --iptables-min-sync-period=10s so that it does at most 1 iptables sync every 10 seconds, thus forcing p50 NetworkProgrammingLatency to hover around (10 seconds minus the length of time it takes to do a single sync).)

last time we analyzed [switching to --iptables-min-sync-period=1s] the latency in iptables improved but CPU increased

Honestly, running kube-proxy with min sync period 10s is just crazy. Like, if a pod crashes, you want to keep sending new connections to its former IP address for another 10 seconds?

If we think kube-proxy uses too much CPU with a shorter sync period then we should fix that. (I think it's not clear how much of the CPU was kube-proxy itself vs kube-proxy plus iptables-restore?)

      "data": {
        "Perc50": 12.5,
        "Perc90": 22.5,
        "Perc99": 24.75
      },
      "unit": "ms",
      "labels": {
        "Metric": "InClusterNetworkLatency"
      }

is the same

That's the one measuring packet latency for pod-to-service-IP-to-pod traffic?

How many services does this job create? We'd probably need at least 10s of thousands for the difference between iptables kube-proxy's rules and nftables kube-proxy's rules to become noticeable.

(Note that for purposes of this metric, you don't even need a large cluster to test that; you can create 1 pod and 10,000 services that all point to that same pod.)

The sync rules in nftables
...
The sync rules in iptables from https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/1778462542213943296 :) that are way better

So that's p95 of sync_proxy_rules_duration_seconds? (aka SyncProxyRulesLatency, aka the time it takes a single syncProxyRules() call to run)

The nftables proxy currently doesn't have the partial-syncing optimizations that iptables has (because we were waiting for the perf jobs before optimizing anything), so this is expected; every sync rewrites every rule, so nftables's sync_proxy_rules_duration_seconds ought to be more comparable to iptables's sync_full_proxy_rules_duration_seconds (note "full")

danwinship · 2024-04-16T14:53:36Z

pkg/features/kube_features.go

@@ -1140,7 +1140,7 @@ var defaultKubernetesFeatureGates = map[featuregate.Feature]featuregate.FeatureS

 	NewVolumeManagerReconstruction: {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.32

-	NFTablesProxyMode: {Default: false, PreRelease: featuregate.Alpha},
+	NFTablesProxyMode: {Default: true, PreRelease: featuregate.Beta},


need to update NFTablesProxyMode comment above with version for beta

danwinship · 2024-04-16T14:56:51Z

cluster/gce/gci/configure-helper.sh

+  if [[ "${KUBE_PROXY_MODE:-}" == "nftables" ]];then
+    params+=" --proxy-mode=nftables"
+  else
+    params+=" --iptables-sync-period=1m --iptables-min-sync-period=10s --ipvs-sync-period=1m --ipvs-min-sync-period=10s"


FWIW, you could set the iptables- and ipvs-specific options regardless of KUBE_PROXY_MODE.

But probably some of this should be merged with the existing [[ "${KUBE_PROXY_MODE:-}" == "ipvs" ]] check? And/or it should do --proxy-mode=${KUBE_PROXY_MODE}.

(Also, typo in the commit message: "configura")

danwinship · 2024-04-16T14:57:35Z

cluster/gce/config-test.sh

-# Optional: Change the kube-proxy implementation. Choices are [iptables, ipvs].
-KUBE_PROXY_MODE=${KUBE_PROXY_MODE:-iptables}
+# Optional: Change the kube-proxy implementation. Choices are [iptables, ipvs, nftables].
+KUBE_PROXY_MODE=${KUBE_PROXY_MODE:-nftables}


(should mark this commit [DO NOT MERGE] or something)

it has to be removed , yes

danwinship · 2024-04-16T14:58:32Z

cmd/kube-proxy/app/server_linux.go

-		o.logger.Info("Using iptables proxy")
-		config.Mode = proxyconfigapi.ProxyModeIPTables
+		o.logger.Info("Using nftables proxy")
+		config.Mode = proxyconfigapi.ProxyModeNFTables


(likewise... or just merge this with the other DNM commit)

danwinship · 2024-04-16T15:01:30Z

cluster/gce/gci/configure-helper.sh

-    params+=" --proxy-mode=nftables"
+    # to avoid the problems caused of adding iptables rules to drop packets with invalid conntrack state
+    # kube-proxy has an option to set the tcpBeLiberal sysctl that solves the problem and is less disruptive
+    # https://github.com/kubernetes/kubernetes/issues/94861ss


Typo in the link (the trailing "ss"), but https://issues.k8s.io/122663#issuecomment-1885024015 is probably a better link anyway.

In particular, we are intentionally requiring that nftables-kube-proxy-based CI set --conntrack-tcp-be-liberal because we want to test that option.

danwinship · 2024-04-18T15:48:42Z

/close
replaced by #124383

k8s-ci-robot · 2024-04-18T15:48:48Z

@danwinship: Closed this PR.

In response to this:

/close
replaced by #124383

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

move nftables proxy to beta

23f8128

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Apr 11, 2024

k8s-ci-robot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Apr 11, 2024

k8s-ci-robot requested a review from danwinship April 11, 2024 14:04

k8s-ci-robot requested review from andrewsykim and bowei April 11, 2024 14:05

aojea mentioned this pull request Apr 11, 2024

scalability job for nftables kubernetes/test-infra#32431

Merged

aojea added 3 commits April 11, 2024 15:41

cluster: configura nftables mode for kube-proxy

48a3e6b

use nftables proxy by default on CI

c354e04

default kube-proxy to nftables

62d970e

aojea force-pushed the nftables_proxy branch from 0e2870d to 62d970e Compare April 11, 2024 15:41

nftables proxy requires conntrack tcp be liberal to be set

fa74ceb

danwinship reviewed Apr 16, 2024

View reviewed changes

k8s-ci-robot closed this Apr 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Nftables proxy #124272

[WIP] Nftables proxy #124272

aojea commented Apr 11, 2024

k8s-ci-robot commented Apr 11, 2024

k8s-ci-robot commented Apr 11, 2024

aojea commented Apr 11, 2024

aojea commented Apr 11, 2024

k8s-ci-robot commented Apr 11, 2024

k8s-ci-robot commented Apr 11, 2024

aojea commented Apr 11, 2024

aojea commented Apr 11, 2024

aojea commented Apr 11, 2024

k8s-ci-robot commented Apr 11, 2024 •

edited

danwinship commented Apr 11, 2024 •

edited

danwinship Apr 16, 2024

danwinship Apr 16, 2024

danwinship Apr 16, 2024

aojea Apr 18, 2024

danwinship Apr 16, 2024

danwinship Apr 16, 2024

danwinship commented Apr 18, 2024

k8s-ci-robot commented Apr 18, 2024

[WIP] Nftables proxy #124272

[WIP] Nftables proxy #124272

Conversation

aojea commented Apr 11, 2024

k8s-ci-robot commented Apr 11, 2024

k8s-ci-robot commented Apr 11, 2024

aojea commented Apr 11, 2024

aojea commented Apr 11, 2024

k8s-ci-robot commented Apr 11, 2024

k8s-ci-robot commented Apr 11, 2024

aojea commented Apr 11, 2024

aojea commented Apr 11, 2024

aojea commented Apr 11, 2024

k8s-ci-robot commented Apr 11, 2024 • edited

danwinship commented Apr 11, 2024 • edited

danwinship Apr 16, 2024

Choose a reason for hiding this comment

danwinship Apr 16, 2024

Choose a reason for hiding this comment

danwinship Apr 16, 2024

Choose a reason for hiding this comment

aojea Apr 18, 2024

Choose a reason for hiding this comment

danwinship Apr 16, 2024

Choose a reason for hiding this comment

danwinship Apr 16, 2024

Choose a reason for hiding this comment

danwinship commented Apr 18, 2024

k8s-ci-robot commented Apr 18, 2024

k8s-ci-robot commented Apr 11, 2024 •

edited

danwinship commented Apr 11, 2024 •

edited