Skip to content

Network testing #28

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 17, 2022
Merged

Network testing #28

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1a4dd47
network performance testing
sionsmith May 12, 2022
6266bcc
network performance testing
sionsmith May 17, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions stable/network-benchmark-testing/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Testing Kubernetes cluster network performance

A standardized benchmark to measure Kubernetes networking performance on multiple host platforms and network stacks.

### Usage
The test suite can be executed via a single Go binary that triggers all the automated testing present on the netperf Docker container.

***NOTE*** A minimum of two Kubernetes worker nodes are necessary for this test. If using minikube you must enable the Docker registry as the image is needed on multiple hosts.

```shell
#create 2 node cluster
minikube start --nodes 2 --cpus=6 --memory=20019 --kubernetes-version=v1.21.0

#enable the docker registry and push image to make available on all nodes
minikube addons enable registry
docker run --rm -it --network=host alpine ash -c "apk add socat && socat TCP-LISTEN:5000,reuseaddr,fork TCP:$(minikube ip):5000" &
docker build ./docker -t netperf:latest
docker tag netperf:latest localhost:5000/netperf:latest
docker push localhost:5000/netperf:latest

#run the tests
cd docker && go run ./launch.go --iterations 1
```

### Kubernetes Network benchmark results
The tests will take some time, after ~35mins you should see the following output.

```shell
│ ALL TESTCASES AND MSS RANGES COMPLETE - GENERATING CSV OUTPUT │
│ MSS , Maximum, 96, 160, 224, 288, 352, 416, 480, 544, 608, 672, 736, 800, 864, 928, 992, 1056, 1120, 1184, 1248, 1312, 1376, 1440, │
│ 1 iperf TCP. Same VM using Pod IP ,43787.000000,40615,41787,41644,34696,38652,40741,39204,33203,39646,24538,43787,32438,27078,16714,33450,41896,35655,41853,40867,41655,40782,41521, │
│ 2 iperf TCP. Same VM using Virtual IP ,45900.000000,37386,36459,37758,40965,41688,38859,39247,41394,37100,35232,40901,35395,36015,41664,45293,44693,40741,42982,41322,44138,40716,45900, │
│ 3 iperf TCP. Remote VM using Pod IP ,27177.000000,26005,24857,26076,24670,24648,24399,24301,23858,23164,21852,24095,24304,25836,24940,25639,25975,22841,24274,24734,26783,24797,27177, │
│ 4 iperf TCP. Remote VM using Virtual IP ,26820.000000,24622,24673,24447,23553,22981,23879,23464,24202,25872,24083,26820,23440,25935,24738,25447,25284,23834,25140,24318,25430,24247,25509, │
│ 5 iperf TCP. Hairpin Pod to own Virtual IP ,43544.000000,38842,39599,38957,39019,39219,38770,39179,39829,41485,39045,43544,39219,38747,42666,40890,41004,37079,37665,37634,41949,37629,42241, │
│ 6 iperf SCTP. Same VM using Pod IP ,2547.000000,2472,2512,2505,2495,2502,2499,2531,1329,1653,1745,1568,1265,175,1259,506,484,2249,2135,2254,2332,2414,2547, │
│ 7 iperf SCTP. Same VM using Virtual IP ,2491.000000,2491,2451,2435,2441,2444,2429,2411,1229,1680,1490,1523,194,300,525,698,1557,1957,2119,2060,2250,2330,2408, │
│ 8 iperf SCTP. Remote VM using Pod IP ,1467.000000,1319,1416,1384,1361,1467,1462,1393,544,1073,1079,921,1020,844,1150,1103,382,1205,1352,1408,1422,1418,1437, │
│ 9 iperf SCTP. Remote VM using Virtual IP ,1540.000000,1438,1409,1216,1364,1534,1451,1540,948,863,1167,995,726,767,736,1240,1231,1191,1400,1441,1476,1441,1517, │
│ 10 iperf SCTP. Hairpin Pod to own Virtual IP ,2356.000000,2236,2344,2300,2356,2309,2149,2146,1257,1476,1424,1378,807,364,954,1447,1409,1866,2025,2143,2210,1517,1131, │
│ 11 iperf UDP. Same VM using Pod IP ,4130.000000,4130, │
│ 12 iperf UDP. Same VM using Virtual IP ,3677.000000,3677, │
│ 13 iperf UDP. Remote VM using Pod IP ,1314.000000,1314, │
│ 14 iperf UDP. Remote VM using Virtual IP ,1281.000000,1281, │
│ 15 netperf. Same VM using Pod IP ,9016.800000,9016.80, │
│ 16 netperf. Same VM using Virtual IP ,0.000000,0, │
│ 17 netperf. Remote VM using Pod IP ,7078.690000,7078.69, │
│ 18 netperf. Remote VM using Virtual IP ,0.000000,0,
```

### Understanding the benchmark data
More information and charting these results [Benchmarking Kubernetes Networking](https://github.com/kubernetes/perf-tests/tree/master/network/benchmarks/netperf#output-raw-csv-data)
31 changes: 31 additions & 0 deletions stable/network-benchmark-testing/docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
ARG GOLANG_VERSION=1.18
FROM golang:${GOLANG_VERSION} as builder
WORKDIR /workspace

COPY nptests/nptest.go nptest.go
COPY go.sum go.sum
COPY go.mod go.mod

RUN go build -o nptests

FROM debian:jessie
ENV LD_LIBRARY_PATH=/usr/local/lib

# install binary and remove cache
RUN apt-get update \
&& apt-get install -y curl wget net-tools gcc make libsctp-dev \
&& rm -rf /var/lib/apt/lists/*
RUN mkdir -p /tmp


# Download and build iperf3 from sources
RUN curl -LO https://downloads.es.net/pub/iperf/iperf-3.1.tar.gz && tar zxf iperf-3.1.tar.gz
RUN cd iperf-3.1 && ./configure --prefix=/usr/local --bindir /usr/local/bin && make && make install

# Download and build netperf from sources
RUN curl -LO https://github.com/HewlettPackard/netperf/archive/netperf-2.7.0.tar.gz && tar -xzf netperf-2.7.0.tar.gz && mv netperf-netperf-2.7.0/ netperf-2.7.0
RUN cd netperf-2.7.0 && ./configure --prefix=/usr/local --bindir /usr/local/bin && make && make install

COPY --from=builder /workspace/nptests /usr/bin/

ENTRYPOINT ["nptests"]
28 changes: 28 additions & 0 deletions stable/network-benchmark-testing/docker/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
module k8s.io/perf-tests/network

go 1.13

require (
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/ghodss/yaml v1.0.1-0.20180820084758-c7ce16629ff4 // indirect
github.com/gogo/protobuf v1.1.2-0.20181010092945-fd322a3c4963 // indirect
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b // indirect
github.com/google/btree v1.0.0 // indirect
github.com/google/gofuzz v0.0.0-20170612174753-24818f796faf // indirect
github.com/googleapis/gnostic v0.2.3-0.20180520015035-48a0ecefe2e4 // indirect
github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 // indirect
github.com/imdario/mergo v0.3.6 // indirect
github.com/json-iterator/go v1.1.6-0.20180914014843-2433035e5132 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.1 // indirect
github.com/peterbourgon/diskv v2.0.2-0.20180312054125-0646ccaebea1+incompatible // indirect
github.com/spf13/pflag v1.0.3 // indirect
github.com/stretchr/testify v1.5.1 // indirect
golang.org/x/oauth2 v0.0.0-20160902055913-3c3a985cb79f // indirect
golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2 // indirect
google.golang.org/appengine v1.6.6 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
k8s.io/api v0.0.0-20181011064954-26c7a45db378
k8s.io/apimachinery v0.0.0-20181011064652-56cf97ad69c7
k8s.io/client-go v0.0.0-20181011105049-9b03088ac34f
)
64 changes: 64 additions & 0 deletions stable/network-benchmark-testing/docker/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/ghodss/yaml v1.0.1-0.20180820084758-c7ce16629ff4 h1:PaTU+9BARuIOAz1ixvps39DJjfq/SxOj3axzIlh7nFo=
github.com/ghodss/yaml v1.0.1-0.20180820084758-c7ce16629ff4/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/gogo/protobuf v1.1.2-0.20181010092945-fd322a3c4963 h1:qxvPMOEHQtKCpBDHjx3NiCnM3NfMEyeA9c72+IljOP4=
github.com/gogo/protobuf v1.1.2-0.20181010092945-fd322a3c4963/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekfRL6a582T4P37/31XEstQ5p58=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/protobuf v1.3.1 h1:YF8+flBXS5eO826T4nzqPrxfhQThhXl0YzfuUPu4SBg=
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/google/btree v1.0.0 h1:0udJVsspx3VBr5FwtLhQQtuAsVc79tTq0ocGIPAU6qo=
github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/gofuzz v0.0.0-20170612174753-24818f796faf h1:+RRA9JqSOZFfKrOeqr2z77+8R2RKyh8PG66dcu1V0ck=
github.com/google/gofuzz v0.0.0-20170612174753-24818f796faf/go.mod h1:HP5RmnzzSNb993RKQDq4+1A4ia9nllfqcQFTQJedwGI=
github.com/googleapis/gnostic v0.2.3-0.20180520015035-48a0ecefe2e4 h1:Z09Qt6AGDtg0cC/YgnX/iymzIqmZf5aasP5JZFxmkNQ=
github.com/googleapis/gnostic v0.2.3-0.20180520015035-48a0ecefe2e4/go.mod h1:sJBsCZ4ayReDTBIg8b9dl28c5xFWyhBTVRp3pOg5EKY=
github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 h1:pdN6V1QBWetyv/0+wjACpqVH+eVULgEjkurDLq3goeM=
github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA=
github.com/imdario/mergo v0.3.6 h1:xTNEAn+kxVO7dTZGu0CegyqKZmoWFI0rF8UxjlB2d28=
github.com/imdario/mergo v0.3.6/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
github.com/json-iterator/go v1.1.6-0.20180914014843-2433035e5132 h1:Y1mRUIuPBTUZfUBQW0V9iItFGJnksjotNXSOzTtxfIo=
github.com/json-iterator/go v1.1.6-0.20180914014843-2433035e5132/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/peterbourgon/diskv v2.0.2-0.20180312054125-0646ccaebea1+incompatible h1:FhnA4iH8T/yYW+AolPONZjGE897wxj3MAzfEbrZkSYw=
github.com/peterbourgon/diskv v2.0.2-0.20180312054125-0646ccaebea1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/spf13/pflag v1.0.3 h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg=
github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.5.1 h1:nOGnQDM7FYENwehXlg/kFVnos3rEvtKTjRvOWSzb6H4=
github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2 h1:VklqNMn3ovrHsnt90PveolxSbWFaJdECFbxSq0Mqo2M=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/net v0.0.0-20190603091049-60506f45cf65 h1:+rhAzEzT3f4JtomfC371qB+0Ola2caSKcY69NUBZrRQ=
golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
golang.org/x/oauth2 v0.0.0-20160902055913-3c3a985cb79f h1:VWt05OS3Al9w09GSPgltoHP90whAHlpik/Bys7HVEDE=
golang.org/x/oauth2 v0.0.0-20160902055913-3c3a985cb79f/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a h1:1BGLXjeY4akVXGgbC9HugT3Jv3hCI0z56oJR5vAMgBU=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2 h1:+DCIGbF/swA92ohVg0//6X2IVY3KZs6p9mix0ziNYJM=
golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
google.golang.org/appengine v1.6.6 h1:lMO5rYAqUxkmaj76jAkRUvt5JZgFymx/+Q5Mzfivuhc=
google.golang.org/appengine v1.6.6/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc=
gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
k8s.io/api v0.0.0-20181011064954-26c7a45db378 h1:lBCdUUoFs8PEMxhQoSNnVXxpKcrNrzuOw4/ofBd+0do=
k8s.io/api v0.0.0-20181011064954-26c7a45db378/go.mod h1:iuAfoD4hCxJ8Onx9kaTIt30j7jUFS00AXQi6QMi99vA=
k8s.io/apimachinery v0.0.0-20181011064652-56cf97ad69c7 h1:l1WAFAYv7ba/kpouP/gXH+l3ZqsRKbGH8F1VMDYmIbM=
k8s.io/apimachinery v0.0.0-20181011064652-56cf97ad69c7/go.mod h1:ccL7Eh7zubPUSh9A3USN90/OzHNSVN6zxzde07TDCL0=
k8s.io/client-go v0.0.0-20181011105049-9b03088ac34f h1:+izcn/RUZFQiB2q/wTE/xgLViQ5iFIZhlQParzg2ha4=
k8s.io/client-go v0.0.0-20181011105049-9b03088ac34f/go.mod h1:7vJpHMYJwNQCWgzmNV+VYUl1zCObLyodBc8nIyt8L5s=
Loading