migrate the internal kube-apiserver to kwok #333

sivchari · 2024-02-06T04:19:15Z

What type of PR is this?

/area simulator

/kind feature
/kind cleanup

What this PR does / why we need it:

Before this PR, we should manage not only kube-scheduler, but also kube-apiserver and etcd.
We wanna focus on kube-scheduler and reduce the time when we maintain various components.

Therefore, this aims to integrate kwok into simulator that simulates Nodes and Clusters..
If this PR is merged, all we have to do only maintain scheduler-simulator.

Which issue(s) this PR fixes:

Fixes #251

Special notes for your reviewer:

/label tide/merge-method-squash

Signed-off-by: sivchari <shibuuuu5@gmail.com>

k8s-ci-robot · 2024-02-06T04:19:25Z

Hi @sivchari. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Signed-off-by: sivchari <shibuuuu5@gmail.com>

wzshiming · 2024-02-08T01:50:37Z

docker-compose-local.yml

-  simulator-etcd:
-    image: quay.io/coreos/etcd:v3.4.26
-    container_name: simulator-etcd
-    restart: always
-    volumes:
-      - simulator-etcd-data:/var/lib/etcd
-    command: etcd --advertise-client-urls http://simulator-etcd:2379 --data-dir /var/lib/etcd --listen-client-urls http://0.0.0.0:2379 --initial-cluster-state new --initial-cluster-token tkn
-    networks:
-      - simulator-internal-network
-volumes:
-  simulator-etcd-data:


I think it's better to create clusters use image, so that the simulator doesn't need to be concerned with the creation and deletion of clusters at all, and only needs to be externally supplied with etcd ports.

simulator-cluster: image: registry.k8s.io/kwok/cluster:v0.5.0-k8s.v1.29.0 container_name: simulator-cluster restart: always ports: - "8080:8080" environment: - KWOK_KUBE_APISERVER_PORT=8080 volumes: - simulator-etcd-data:/var/lib/etcd - ./kwokctl.yaml:/root/.kwok/kwok.yaml

# kwokctl.yaml kind: KwokctlConfiguration apiVersion: config.kwok.x-k8s.io/v1alpha1 options: etcdPort: 2379 etcdPrefix: /kube-scheduler-simulator disableKubeScheduler: true componentsPatches: - name: kube-apiserver extraArgs: - key: cors-allowed-origins value: ^*$

sivchari#1

Sorry for the clutter. Although I tried to fix these codes, it couldn't catch the watchResources response. it's suspended and I don't have no idea why it's caused. Do you have some idea ?

wzshiming · 2024-02-08T01:56:20Z

/ok-to-test

Signed-off-by: sivchari <shibuuuu5@gmail.com>

sivchari · 2024-02-09T03:50:25Z

@sanposhiho
I'm ready for review.

@wzshiming
Thanks for your so kind support.

sivchari · 2024-02-09T03:50:49Z

/cc @sanposhiho
/cc @196Ikuchil

Signed-off-by: sivchari <shibuuuu5@gmail.com>

sivchari · 2024-03-15T12:19:59Z

@sanposhiho @wzshiming
I'm sorry making you late. I wasn't able to deal with it.
Since I've handled it, please review some new changes. Thanks.

sanposhiho · 2024-03-15T12:50:18Z

simulator/docs/how-it-works.md

@@ -5,12 +5,15 @@ This page describes how the simulator works.
 ### 0. starts the simulator.

 The simulator server works with the following:


Suggested change

The simulator server works with the following:

The simulator server contains the following components:

sanposhiho · 2024-03-15T12:51:23Z

simulator/docs/how-it-works.md

+- [kube-apiserver](kube-apiserver.md)
+- etcd
 - scheduler
 - controllers for core resources
 - [HTTP server](api.md) 


Suggested change

- [kube-apiserver](kube-apiserver.md)

- etcd

- scheduler

- controllers for core resources

- [HTTP server](api.md)

- scheduler

- [HTTP server](api.md)

sanposhiho · 2024-03-15T12:51:44Z

simulator/docs/kube-apiserver.md

@@ -1,5 +1,11 @@
 ## Kube-apiserver


I think we can just remove this page

sanposhiho · 2024-03-15T12:54:20Z

simulator/docs/how-it-works.md

+In advance, the simulator needs to launch etcd, controllers and kube-apiserver outside.
+[KWOK](https://github.com/kubernetes-sigs/kwok) can launch these components all at once, thus we recommend using it.
+When the simulator server starts, it will start scheduler and HTTP server.


Suggested change

In advance, the simulator needs to launch etcd, controllers and kube-apiserver outside.

[KWOK](https://github.com/kubernetes-sigs/kwok) can launch these components all at once, thus we recommend using it.

When the simulator server starts, it will start scheduler and HTTP server.

In advance, the simulator needs to launch etcd, controller-manager and kube-apiserver outside.

We recommend using [KWOK](https://github.com/kubernetes-sigs/kwok), see [docker-compose.yml](../../docker-compose.yml) to know how we wire things up.

sanposhiho · 2024-03-15T12:57:07Z

simulator/docs/how-it-works.md

-When the simulator server starts, it will start these components with server.
+In advance, the simulator needs to launch etcd, controllers and kube-apiserver outside.
+[KWOK](https://github.com/kubernetes-sigs/kwok) can launch these components all at once, thus we recommend using it.
+When the simulator server starts, it will start scheduler and HTTP server.

 ### 1. users request creating resource.


Kwok should support most of resources and we no longer need to list supported resources. We can simplify this section like:

### 1. users request creating resource. Users can create resources below by communicating with kube-apiserver of Kwok via any clients (e.g., kubectl, kwokctl, k8s client library or [Web UI](../../web)) ### 2. the scheduler schedules a new pod ...

sanposhiho · 2024-03-15T13:04:01Z

simulator/docs/how-it-works.md

Similarly, Please modify README too.

sanposhiho · 2024-03-15T13:19:10Z

simulator/simulator.go

-	time.Sleep(1 * time.Second)
+	ctx, cancel := context.WithTimeout(context.Background(), 1*time.Minute)
+	defer cancel()
+	wait.UntilWithContext(ctx, func(ctx context.Context) {


Sorry, I misled in a previous comment.
We should use PollUntilContextTimeout rather, otherwise wait.UntilWithContext wouldn't panic or error out if reaching timeout seemingly.
https://pkg.go.dev/k8s.io/apimachinery/pkg/util/wait#PollUntilContextTimeout

Signed-off-by: sivchari <shibuuuu5@gmail.com>

sanposhiho · 2024-03-17T01:55:20Z

README.md

@@ -41,12 +41,12 @@ We have several components:



Can we update Getting started section like this otherwise people cannot run up the simulator correctly until a new release is cut.

## Getting started git clone git@github.com:kubernetes-sigs/kube-scheduler-simulator.git cd kube-scheduler-simulator # switch to the version you want to use. git switch simulator/v0.1.1 # pull images from the registry and run up all components make docker_up # All things up!

I see. I'd revise it.

sanposhiho · 2024-03-17T02:06:15Z

README.md

+You can create any resources by communication with kube-apiserver outside via any clients (e.g. kubectl, k8s client library, or web UI described next).
 And when you create Pods, 
 Pods will be scheduled by the [debuggable scheduler](./simulator/docs/debuggable-scheduler.md),


nit

Suggested change

You can create any resources by communication with kube-apiserver outside via any clients (e.g. kubectl, k8s client library, or web UI described next).

And when you create Pods,

Pods will be scheduled by the [debuggable scheduler](./simulator/docs/debuggable-scheduler.md),

Simulator runs with a fake cluster powered by [KWOK](https://github.com/kubernetes-sigs/kwok)

You can create any resources in KWOK cluster via any clients (e.g. kubectl, k8s client library, or web UI described next).

And when you create Pods,

Pods will be scheduled by the [debuggable scheduler](./simulator/docs/debuggable-scheduler.md),

sanposhiho · 2024-03-17T02:35:29Z

simulator/simulator.go

+		_, err := client.CoreV1().Namespaces().Get(context.Background(), "kube-system", metav1.GetOptions{})
+		if err != nil {
+			klog.Infof("waiting for kube-system namespace to be ready: %v", err)
+			return false, err


I guess if we here return err, it'd immediately fail without retry. Let's just return nill as we log error in L65.

Suggested change

return false, err

return false, nil

sanposhiho · 2024-03-17T02:46:58Z

simulator/config/config.go

+	url := os.Getenv("KUBE_APISERVER_URL")
+	if url == "" && configYaml.KubeAPIServerURL != "" {
+		return configYaml.KubeAPIServerURL
+	}
 	p := os.Getenv("KUBE_API_PORT")


Let's remove L145-160, but without removing KubeAPIPort and KubeAPIHost from simulator/config/v1alpha1/types.go. We can just ignore deprecated envs (as I said before, removing them would be a breaking change that we should avoid though

Signed-off-by: sivchari <shibuuuu5@gmail.com>

sivchari · 2024-03-18T05:02:32Z

I've updated PR, thanks.

wzshiming

Currently, the image startup script is hardcoded, the only way to set this is through the environment variable.
and the next release (kubernetes-sigs/kwok#999) will support kubeApiserverInsecurePort to instead of this

docker-compose-local.yml

wzshiming · 2024-03-18T07:24:22Z

kwok.yaml

+kind: KwokctlConfiguration
+apiVersion: config.kwok.x-k8s.io/v1alpha1
+options:
+  kubeApiserverPort: 3131


Suggested change

kubeApiserverPort: 3131

Signed-off-by: sivchari <shibuuuu5@gmail.com>

sivchari · 2024-03-18T09:47:21Z

@wzshiming
Thanks!! I've checked the script of kwok's entry point and the plan of next release.
Once, I decided to use a env to connect the cluster in KWOK.

docker-compose-local.yml

docker-compose.yml

kwok.yaml

sanposhiho

Two doc changes, otherwise LGTM

sanposhiho · 2024-03-26T00:59:51Z

simulator/docs/environment-variables.md

@@ -28,15 +33,20 @@ origin for `CORS_ALLOWED_ORIGIN_LIST`.
 resources". This variable is used to find Kubeconfig required to
 access your cluster for importing resources to scheduler simulator.

+`KUBE_APISEVER_URL`: This is the URL of kube-apiserver which the
+simulator uses. This variable is used to connect to external kube-apiserver.
+
 `KUBE_API_HOST`: This is the host of kube-apiserver which the


Sorry, on second thought, I don't think we need to keep the mention of KUBE_API_HOST and KUBE_API_PORT.
Can we just remove L8-L12 and L36-L45?

sanposhiho · 2024-03-26T01:02:29Z

simulator/docs/environment-variables.md

We forget about the doc in:
https://github.com/kubernetes-sigs/kube-scheduler-simulator/blob/f0faa06ec7773723073ad04c4bb3383dbbd8d3ff/simulator/docs/simulator-server-config.md

Can you

remove the mention to kubeApiHost and kubeApiPort

mention kubeApiServerUrl

over there?

Signed-off-by: sivchari <shibuuuu5@gmail.com>

sivchari · 2024-03-26T01:08:46Z

@sanposhiho
I've revised it, thanks.

sanposhiho

/lgtm
/approve

@sivchari This is epic, thanks for your all effort!

k8s-ci-robot · 2024-03-26T01:10:36Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sanposhiho, sivchari

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [sanposhiho]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sanposhiho · 2024-03-26T01:13:59Z

Also thanks @wzshiming for your all support! It'd be hard to achieve if you weren't here.

sivchari added 3 commits January 30, 2024 16:28

install: e2e-framework

f990efe

Signed-off-by: sivchari <shibuuuu5@gmail.com>

poc: use kwok

6095ae6

Signed-off-by: sivchari <shibuuuu5@gmail.com>

simulator: specify etcd/api-server port

6c347ec

Signed-off-by: sivchari <shibuuuu5@gmail.com>

k8s-ci-robot requested review from 196Ikuchil and Huang-Wei February 6, 2024 04:19

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Feb 6, 2024

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Feb 6, 2024

sivchari added 2 commits February 6, 2024 13:23

disable: kube-scheduler

a02fc4f

Signed-off-by: sivchari <shibuuuu5@gmail.com>

use: kubectl proxy

258f115

Signed-off-by: sivchari <shibuuuu5@gmail.com>

wzshiming reviewed Feb 8, 2024

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Feb 8, 2024

sivchari added 6 commits February 9, 2024 12:46

update: container settings

ba7cc8a

Signed-off-by: sivchari <shibuuuu5@gmail.com>

update: module dependencies

482ae82

Signed-off-by: sivchari <shibuuuu5@gmail.com>

update: simulator configuration

2545f2c

Signed-off-by: sivchari <shibuuuu5@gmail.com>

revert: Dockerfile

285cf1c

Signed-off-by: sivchari <shibuuuu5@gmail.com>

remove: unnecessary config param

51cb468

Signed-off-by: sivchari <shibuuuu5@gmail.com>

clean up kwok codes

550bd4a

Signed-off-by: sivchari <shibuuuu5@gmail.com>

k8s-ci-robot requested a review from sanposhiho February 9, 2024 03:50

sivchari force-pushed the use-kwok branch from 10b6860 to 550bd4a Compare February 9, 2024 03:51

sivchari added 2 commits March 15, 2024 21:09

Define the etcd prefix and use it

d637020

Signed-off-by: sivchari <shibuuuu5@gmail.com>

Use wait package to check whether cluster is ready

9bd672f

Signed-off-by: sivchari <shibuuuu5@gmail.com>

sanposhiho reviewed Mar 15, 2024

View reviewed changes

sivchari added 3 commits March 15, 2024 22:40

simlify README.md

031d4c8

Signed-off-by: sivchari <shibuuuu5@gmail.com>

docs: modify some sentences

0aca48b

Signed-off-by: sivchari <shibuuuu5@gmail.com>

use PollUntil instead of Until

ec64eb7

Signed-off-by: sivchari <shibuuuu5@gmail.com>

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Mar 15, 2024

fix errcheck

de0051b

Signed-off-by: sivchari <shibuuuu5@gmail.com>

sanposhiho reviewed Mar 17, 2024

View reviewed changes

sivchari added 3 commits March 18, 2024 13:59

add more information to README.md

6c8130f

Signed-off-by: sivchari <shibuuuu5@gmail.com>

return nil even if an error is caused

6a502ff

Signed-off-by: sivchari <shibuuuu5@gmail.com>

ignore KUBE_API_PORT and KUBE_API_HOST

89eef35

Signed-off-by: sivchari <shibuuuu5@gmail.com>

wzshiming reviewed Mar 18, 2024

View reviewed changes

add env for kubeapi-server to connect to cluster in KWOK

f0faa06

Signed-off-by: sivchari <shibuuuu5@gmail.com>

sanposhiho reviewed Mar 22, 2024

View reviewed changes

docker-compose-local.yml Show resolved Hide resolved

docker-compose.yml Show resolved Hide resolved

kwok.yaml Show resolved Hide resolved

sanposhiho reviewed Mar 26, 2024

View reviewed changes

revise documents

f73674a

Signed-off-by: sivchari <shibuuuu5@gmail.com>

sanposhiho approved these changes Mar 26, 2024

View reviewed changes

k8s-ci-robot assigned sanposhiho Mar 26, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 26, 2024

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 26, 2024

k8s-ci-robot merged commit 296766e into kubernetes-sigs:master Mar 26, 2024
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

migrate the internal kube-apiserver to kwok #333

migrate the internal kube-apiserver to kwok #333

sivchari commented Feb 6, 2024 •

edited

Loading

k8s-ci-robot commented Feb 6, 2024

wzshiming Feb 8, 2024 •

edited

Loading

sivchari Feb 8, 2024 •

edited

Loading

wzshiming commented Feb 8, 2024

sivchari commented Feb 9, 2024

sivchari commented Feb 9, 2024

sivchari commented Mar 15, 2024

sanposhiho Mar 15, 2024

sanposhiho Mar 15, 2024

sanposhiho Mar 15, 2024

sanposhiho Mar 15, 2024

sanposhiho Mar 15, 2024

sanposhiho Mar 15, 2024

sanposhiho Mar 15, 2024

sanposhiho Mar 17, 2024

sivchari Mar 18, 2024

sanposhiho Mar 17, 2024 •

edited

Loading

sanposhiho Mar 17, 2024

sanposhiho Mar 17, 2024

sivchari commented Mar 18, 2024

wzshiming left a comment •

edited

Loading

wzshiming Mar 18, 2024

sivchari commented Mar 18, 2024

sanposhiho left a comment

sanposhiho Mar 26, 2024 •

edited

Loading

sanposhiho Mar 26, 2024

sivchari commented Mar 26, 2024

sanposhiho left a comment

k8s-ci-robot commented Mar 26, 2024

sanposhiho commented Mar 26, 2024 •

edited

Loading

		@@ -5,12 +5,15 @@ This page describes how the simulator works.
		### 0. starts the simulator.

		The simulator server works with the following:

	The simulator server works with the following:
	The simulator server contains the following components:

migrate the internal kube-apiserver to kwok #333

migrate the internal kube-apiserver to kwok #333

Conversation

sivchari commented Feb 6, 2024 • edited Loading

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

k8s-ci-robot commented Feb 6, 2024

wzshiming Feb 8, 2024 • edited Loading

Choose a reason for hiding this comment

sivchari Feb 8, 2024 • edited Loading

Choose a reason for hiding this comment

wzshiming commented Feb 8, 2024

sivchari commented Feb 9, 2024

sivchari commented Feb 9, 2024

sivchari commented Mar 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanposhiho Mar 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sivchari commented Mar 18, 2024

wzshiming left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sivchari commented Mar 18, 2024

sanposhiho left a comment

Choose a reason for hiding this comment

sanposhiho Mar 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sivchari commented Mar 26, 2024

sanposhiho left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Mar 26, 2024

sanposhiho commented Mar 26, 2024 • edited Loading

sivchari commented Feb 6, 2024 •

edited

Loading

wzshiming Feb 8, 2024 •

edited

Loading

sivchari Feb 8, 2024 •

edited

Loading

sanposhiho Mar 17, 2024 •

edited

Loading

wzshiming left a comment •

edited

Loading

sanposhiho Mar 26, 2024 •

edited

Loading

sanposhiho commented Mar 26, 2024 •

edited

Loading