scale-test: Measure APIServer SLOs

[hakuna-matatah]

- This PR makes CL2 to install prometheus stack locally before kicking off the load test
- Update go deps from kubernetes-sigs/kubetest2#244

• Also update the worker node architecture to amd64 for prometheus and grafana nodes to come up successfully.
• Also this https://github.com/kubernetes/kops/pull/15963#issuecomment-1760718194
• Also this https://github.com/kubernetes/kops/pull/15963#issuecomment-1766864001
• Also this https://github.com/kubernetes/kops/pull/15963#issuecomment-1767112397
• Clean up to make the script to only run CL2 and not conformance

[k8s-ci-robot]

Hi @hakuna-matatah. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dims]

/ok-to-test
/approve
/lgtm

[hakman]

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

[hakman]

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

[hakman]

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

[hakman]

@hakuna-matatah

kubetest2 kops -v=2 --cloud-provider=aws --cluster-name=e2e-ff02749ef8-a423a.test-cncf-aws.k8s.io --kops-binary-path=/home/prow/go/src/k8s.io/bin/kops --admin-access=0.0.0.0/0 --env=KOPS_FEATURE_FLAGS=ClusterAddons, --validation-wait=45m --test=clusterloader2 --kubernetes-version=v1.28.2 --enable-prometheus-server=true --prometheus-pvc-storage-class=gp2 --provider=aws --repo-root=/home/prow/go/src/k8s.io/perf-tests --test-configs=/home/prow/go/src/k8s.io/perf-tests/clusterloader2/testing/load/config.yaml --test-configs=/home/prow/go/src/k8s.io/perf-tests/clusterloader2/testing/huge-service/config.yaml --test-configs=/home/prow/go/src/k8s.io/perf-tests/clusterloader2/testing/access-tokens/config.yaml --kube-config=/tmp/kubeconfig.yL6RMrPS9
Error: unknown flag: --enable-prometheus-server

[dims]

@hakman kubernetes-sigs/kubetest2#244

[hakman]

Thanks @dims! 😄
@hakuna-matatah Could you also check what limit was hit when both pre-submit and periodic were running at the same time this morning?

[upodroid]

one more thing, you'll run in to this problem if you try to bump the kubetest2 deps

kubernetes-sigs/boskos#173

[hakuna-matatah]

Thanks @dims! 😄 @hakuna-matatah Could you also check what limit was hit when both pre-submit and periodic were running at the same time this morning?

What limits are you referring to ? EC2 vcpus ?

[hakman]

What limits are you referring to ? EC2 vcpus ?

I suspect that there is a limit on network interfaces (for sure not vCPUs).
Nodes are allocated, but some pods are still in pending:

VALIDATION ERRORS
KIND	NAME						MESSAGE
Machine	i-004735166086eb1a8				machine "i-004735166086eb1a8" has not yet joined cluster
Machine	i-04d472f0ade8444a2				machine "i-04d472f0ade8444a2" has not yet joined cluster
Machine	i-080a7e812c07c8b5f				machine "i-080a7e812c07c8b5f" has not yet joined cluster
Machine	i-08765c46553ad7ec5				machine "i-08765c46553ad7ec5" has not yet joined cluster
Machine	i-0cbcdc04bc9c43d8e				machine "i-0cbcdc04bc9c43d8e" has not yet joined cluster
Machine	i-0d78f9a4fd2ac19f1				machine "i-0d78f9a4fd2ac19f1" has not yet joined cluster
Machine	i-0da80da3246b8bf33				machine "i-0da80da3246b8bf33" has not yet joined cluster
Machine	i-0dcf5f85fc27ec1ae				machine "i-0dcf5f85fc27ec1ae" has not yet joined cluster
Machine	i-0ec9351e879207a21				machine "i-0ec9351e879207a21" has not yet joined cluster
Machine	i-0f06977ad9681e2a0				machine "i-0f06977ad9681e2a0" has not yet joined cluster
Machine	i-0f7a95e2c34a855f4				machine "i-0f7a95e2c34a855f4" has not yet joined cluster
Node	i-040f427732d5daf27				node "i-040f427732d5daf27" of role "node" is not ready
Pod	kube-system/aws-node-r8dxf			system-node-critical pod "aws-node-r8dxf" is pending
Pod	kube-system/aws-node-tm696			system-node-critical pod "aws-node-tm696" is pending
Pod	kube-system/aws-node-trrdn			system-node-critical pod "aws-node-trrdn" is pending
Pod	kube-system/aws-node-wqrb6			system-node-critical pod "aws-node-wqrb6" is pending
Pod	kube-system/ebs-csi-node-cknr6			system-node-critical pod "ebs-csi-node-cknr6" is pending
Pod	kube-system/ebs-csi-node-kj8r5			system-node-critical pod "ebs-csi-node-kj8r5" is pending
Pod	kube-system/ebs-csi-node-qrgf5			system-node-critical pod "ebs-csi-node-qrgf5" is pending
Pod	kube-system/ebs-csi-node-snsfc			system-node-critical pod "ebs-csi-node-snsfc" is pending
Pod	kube-system/kube-proxy-i-004735166086eb1a8	system-node-critical pod "kube-proxy-i-004735166086eb1a8" is pending

This is the run:
https://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws-scale-amazonvpc-using-cl2/1706544867737341952/

[hakuna-matatah]

Apparently tests are now failing because kube-proxy service is looking for label selector [component](https://github.com/kubernetes/perf-tests/blob/master/clusterloader2/pkg/prometheus/manifests/default/kube-proxy-service.yaml#L2) by default. But kops creates kube-proxy with label k8s-app which makes kube-proxy service to not match with any of the kube-proxy pod endpoints , which leads the prometheus to not be able to scrape kubeproxy and thus leading to CL2 tests failures with errors NetworkProgrammingLatency gathering error: got unexpected number of samples: 0 .

Resolved this issue in the last latest commit on this PR by passing env variable for kube-proxy selector key

[hakuna-matatah]

/test presubmit-kops-aws-small-scale-amazonvpc-using-cl2

[hakuna-matatah]

Now that kube-proxy endpoints are being detected due to above fix in the comment , ran into another blocker, i.e; prometheus is not able to get data from metrics endpoint of kube-proxy .

[hakuna-matatah]

Now that kube-proxy endpoints are being detected due to above fix in the comment , ran into another blocker, i.e; prometheus is not able to get data from metrics endpoint of kube-proxy .

It appears that MetricsBindAddress on kube-proxy is defaulted to localhost which is why I was able to get metrics locally but prometheus could not scrape at node level port, see below for more details. I'm planning to set the

HTTP/1.1 200 OK
Content-Type: text/plain; version=0.0.4; charset=utf-8
Process-Start-Time-Unix: 1697069093
Date: Tue, 17 Oct 2023 20:13:53 GMT
Transfer-Encoding: chunked

# HELP aggregator_discovery_aggregation_count_total [ALPHA] Counter of number of times discovery was aggregated
# TYPE aggregator_discovery_aggregation_count_total counter
aggregator_discovery_aggregation_count_total 0
# HELP apiserver_audit_event_total [ALPHA] Counter of audit events generated and sent to the audit backend.
# TYPE apiserver_audit_event_total counter
apiserver_audit_event_total 0
# HELP apiserver_audit_requests_rejected_total [ALPHA] 

I have set the kubeProxy metricsBindAddress to spec.kubeProxy.metricsBindAddress=0.0.0.0:10249" and now it looks like it is able to scrape metrics endpoint locally.

[hakuna-matatah]

/test presubmit-kops-aws-small-scale-amazonvpc-using-cl2

[hakuna-matatah]

/test presubmit-kops-aws-small-scale-amazonvpc-using-cl2

[k8s-ci-robot]

@hakuna-matatah: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
presubmit-kops-aws-scale-amazonvpc-using-cl2	211d111	link	true	/test presubmit-kops-aws-scale-amazonvpc-using-cl2

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[hakuna-matatah]

• Successfully tested the changes in this PR and tests have passed here in this run - https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kops/15963/presubmit-kops-aws-small-scale-amazonvpc-using-cl2/1714377536231706624
• Need to sync these changes before we run again the tests on this PR as expected because controlplane instance-type was being overridden due to env variable in this line which gives preference to changes from test-infra repo

[hakuna-matatah]

/retest

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dims, hakman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [hakman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment