Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

operator init operations may need bigger timeout #3935

Closed
Vacant2333 opened this issue Aug 14, 2023 · 12 comments · Fixed by #3944
Closed

operator init operations may need bigger timeout #3935

Vacant2333 opened this issue Aug 14, 2023 · 12 comments · Fixed by #3944
Assignees
Labels
kind/question Indicates an issue that is a support question.

Comments

@Vacant2333
Copy link
Contributor

What happened:
When deploying Karmada using the operator, it's important to note that for the three components - karmada-etcd, karmada-apiserver, and karmada-aggregated-apiserver - their timeout is set to 30 seconds. In my environment (and even in most environments), if the images are not pre-pulled beforehand, the allotted time might not be sufficient. As a result, deployment could fail due to timeouts, necessitating the deletion of the Karmada resource before reattempting the deployment.
What you expected to happen:
Install success at once
How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:
83d4c1419dcaa875d2e74015433b0c5d
0a71d6f83ce6c6e7d54441112e12d838
7bf89d25c6f9cd31f726b821206899e9
image

@Vacant2333 Vacant2333 added the kind/bug Categorizes issue or PR as related to a bug. label Aug 14, 2023
@XiShanYongYe-Chang
Copy link
Member

/cc @calvin0327 @Poor12 @lonelyCZ
Hi, can you help take a look? Can we modify this time to be configurable?

@Vacant2333
Copy link
Contributor Author

/kind question
/assign

@karmada-bot karmada-bot added the kind/question Indicates an issue that is a support question. label Aug 14, 2023
@XiShanYongYe-Chang
Copy link
Member

/remove-kind bug

@karmada-bot karmada-bot removed the kind/bug Categorizes issue or PR as related to a bug. label Aug 14, 2023
@RainbowMango
Copy link
Member

Which version of karmadactl we are talking about? I remember we have introduced a flag(--wait-component-ready-timeout) for this:

-bash-5.0# karmadactl init --help 
Install the Karmada control plane in a Kubernetes cluster.

Options:
    --wait-component-ready-timeout=120:
	Wait for karmada component ready timeout. 0 means wait forever

Usage:
  karmadactl init [options]

Use "karmadactl options" for a list of global command-line options (applies to all commands).

Please try with the karmadactl on master.

@Vacant2333
Copy link
Contributor Author

Which version of karmadactl we are talking about? I remember we have introduced a flag(--wait-component-ready-timeout) for this:

-bash-5.0# karmadactl init --help 
Install the Karmada control plane in a Kubernetes cluster.

Options:
    --wait-component-ready-timeout=120:
	Wait for karmada component ready timeout. 0 means wait forever

Usage:
  karmadactl init [options]

Use "karmadactl options" for a list of global command-line options (applies to all commands).

Please try with the karmadactl on master.

its install by operator, not karmadactl
https://github.com/karmada-io/karmada/blob/master/operator/README.md

@RainbowMango
Copy link
Member

oh, sorry my bad.

@Vacant2333
Copy link
Contributor Author

may we can update to 120s, i think its enough at most situations

@chaosi-zju
Copy link
Member

@RainbowMango

I have experienced similar problems in install by helm chart (#3905 (comment)), the root cause is pulling the mirror image is too slow.

From a user perspective, it greatly affects the installation experience. This will cause users to dislike our slow installation process, and they may not know that it is actually due to their poor network that the image pulling process is slow. In fact, our installation process is very simple and fast.

I tend to separate pulling images from other installation operations, that is, pulling images first before performing other operations. And, user will get a install log like:

start pulling images:
start pull image karmada-xxxx ....... done
start pull image karmada-yyyy ....... done
start pull image kube-xxxx  ......... done
....
start installation:
....

As for the problem this issue mentioned, we just don't give timeout limit to image pulling process, and keep previous timeout to other install operation.

Thus, even though the sum time of installation could be unchanged, users can see at a glance that the slow installation is mainly due to slow image pulling. Users will feel that in addition to the image pulling process, the installation is fast and smooth.

@RainbowMango
Copy link
Member

may we can update to 120s, i think its enough at most situations

@Vacant2333 Can you show us which pieces of code we need to update?
I agree that keeping the default timeout consistent between CLI/operator/chart.

@RainbowMango
Copy link
Member

I tend to separate pulling images from other installation operations, that is, pulling images first before performing other operations. And, user will get a install log like:

How to separate pulling images?

@Vacant2333
Copy link
Contributor Author

I tend to separate pulling images from other installation operations, that is, pulling images first before performing other operations. And, user will get a install log like:

How to separate pulling images?

you can check my pr, there is 3 task need to be update

@chaosi-zju
Copy link
Member

I tend to separate pulling images from other installation operations, that is, pulling images first before performing other operations. And, user will get a install log like:

How to separate pulling images?

This requires some exploration, simply thought, if it is just based on Kind, it can be done like this:

docker pull registry.k8s.io/etcd:3.5.9-0
kind load docker-image registry.k8s.io/etcd:3.5.9-0 --name karmada-host

But this is not a generic approach, for not all k8s is based on Kind, generic approach may need execute crictl pull on the node of karmada-host, like:

docker exec -it karmada-host-control-plane bash
root@karmada-host-control-plane:/# crictl --debug pull registry.k8s.io/etcd:3.5.9-0   
DEBU[0000] get image connection                         
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:registry.k8s.io/etcd:3.5.9-0,Annotations:map[string]string{},},Auth:nil,SandboxConfig:nil,} 
DEBU[0078] PullImageResponse: &PullImageResponse{ImageRef:sha256:73deb9a3f702532592a4167455f8bf2e5f5d900bcc959ba2fd2d35c321de1af9,} 
Image is up to date for sha256:73deb9a3f702532592a4167455f8bf2e5f5d900bcc959ba2fd2d35c321de1af9

But this is just a bit of thinking and requires more exploration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/question Indicates an issue that is a support question.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants