new Kubernetes cluster goes into an error state

[Stanley2911]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Kubernetes

CLOUDSTACK VERSION 4.18.0

CONFIGURATION

KVM on Rocky Linux 8.7

OS / ENVIRONMENT

SUMMARY

Whenever I am provisioning a new Kubernetes cluster, the cluster goes into an error state. The error obtained on GUI is given below

Error
Failed to setup Kubernetes cluster : k8s as it does not have desired number of nodes in ready state

Upon further inspection, the cluster has both controller & worker nodes in running state.

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[kiranchavala]

@Stanley2911 what is the kubernetes version you are trying to launch ?

[Stanley2911]

v1.23, v.1.25, v 1.26, I am getting same error on all versions Sent from Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: kiranchavala ***@***.***> Sent: Monday, July 10, 2023 9:38:46 AM To: apache/cloudstack ***@***.***> Cc: Stanley2911 ***@***.***>; Mention ***@***.***> Subject: Re: [apache/cloudstack] kubernetes issue (Issue #7734) @Stanley2911<https://github.com/Stanley2911> what is the kubernetes version you are trying to launch ? — Reply to this email directly, view it on GitHub<#7734 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BAB7JIBA2I5GFL6YGVPJH7TXPN545ANCNFSM6AAAAAA2B4Z6HQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[rohityadavcloud]

@Stanley2911 thanks for reporting the issue, do you have overprovisioned RAM in your env? Can you try 1.26.6 or 1.27.3 isos from https://download.cloudstack.org/cks/ ?

[Stanley2911]

Hi Rohit,

Thanks a lot for your replies. I ll try to add v1.26.6 & 1.27.3 as well and get back to you.

[Stanley2911]

Hi Rohit,

Tried with both ISOs v1.26.6 & v1.27.3 as well. I am getting the same error. Screenshots are shared for your reference

[kiranchavala]

@Stanley2911 what type of network are you using "stank8s2network" is it isolated network or a shared network ?

[Stanley2911]

Hello Kiranchavala,

Thanks a lot for your response.
The stank8s2network is provisioned as an Isolated network. The screenshot is attached below

[kiranchavala]

@Stanley2911

1. Can you check if you are able to telnet to public IP of the CKS cluster on port 2222 , 6443.

You can get the publc ip address from the UI ( screenshots attached)

2. Can you try to ssh into the controller node and worker node ( using the ssh key)

Master node
ssh -i <ssh-private.key > -p 2222 cloud@Public ip address of Virtual Router

worker node
ssh -i <ssh-private.key > -p 2223 cloud@Public ip address of Virtual Router

3. Once you are able to ssh into the control node and worker node

Please check the /var/log/daemon.log

[Stanley2911]

I ll check & logs & get back to you. Thanks Sent from Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: kiranchavala ***@***.***> Sent: Wednesday, July 12, 2023 10:59:27 AM To: apache/cloudstack ***@***.***> Cc: Stanley2911 ***@***.***>; Mention ***@***.***> Subject: Re: [apache/cloudstack] kubernetes issue (Issue #7734) @Stanley2911<https://github.com/Stanley2911> 1. Can you check if you are able to telnet to public IP of the CKS cluster on port 2222 , 6443. You can get the publc ip address from the UI ( screenshots attached) [Screenshot 2023-07-12 at 10 47 35 AM]<https://user-images.githubusercontent.com/1401014/252862350-08e423f1-c3d9-446b-8131-7f48f070dcb5.png> [Screenshot 2023-07-12 at 10 47 49 AM]<https://user-images.githubusercontent.com/1401014/252862390-a8c8ffc2-c53f-44aa-ad09-f2f27678d883.png> 1. Can you try to ssh into the controller node and worker node ( using the ssh key) Master node ssh -i <ssh-private.key > -p 2222 ***@***.*** ip address of Virtual Router worker node ssh -i <ssh-private.key > -p 2223 ***@***.*** ip address of Virtual Router [Screenshot 2023-07-12 at 10 48 05 AM]<https://user-images.githubusercontent.com/1401014/252862414-31f12fa3-2040-405e-8501-91c719044ef2.png> 1. Once you are able to ssh into the control node and worker node Please check the /var/log/daemon.log — Reply to this email directly, view it on GitHub<#7734 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BAB7JIGTT4LYT2IEVL46LQTXPYY3PANCNFSM6AAAAAA2B4Z6HQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Stanley2911]

Hi kiranchavala,

I have uploaded the both controller & worker node log files.
Please let me know what can be the issue.

cloud@stank8s2-control-1894435c37b ~.txt

worker.txt

[Stanley2911]

Hi Kiranchavala,

Thanks for your help.
Apologies. The previously uploaded worker node file is empty.
The worker node log file is given below

worker_node.txt

[kiranchavala]

@Stanley2911 could you please check if there are older daemon logs

From the worker node, i need the logs for the date jul 11 as the maste node logs shows the same date

current worker node logs are for the jul 13 th