Change CloudStack-Secret in CloudStack Kubernetes Provider

[ahmadamirahmadi007]

ISSUE TYPE

• Other

COMPONENT NAME

CloudStack Kubernetes Provider

CLOUDSTACK VERSION

4.17.2

OS / ENVIRONMENT

ubuntu 20.04

SUMMARY

CloudStack Internally Use This cloud-config in CloudStack Kubernetes Provider and this happen when kubernetes cluster create from my kubernetes.iso template file. i want change them to my custom config-file but i dont know how cloudstack set it and how can i change it.

[Global]
api-url =
api-key =
secret-key =
project-id = <CloudStack Project UUID (optional)>
zone = <CloudStack Zone Name (optional)>
ssl-no-verify = <Disable SSL certificate validation: true or false (optional)>

STEPS TO REPRODUCE

i created iso from this shell script:
https://github.com/apache/cloudstack/blob/main/scripts/util/create-kubernetes-binaries-iso.sh

EXPECTED RESULTS

i want change cloudstack-secret to my custome file

ACTUAL RESULTS

i cant change it

[shwstppr]

@ahmadamirahmadi007 from what I can understand from the code the cloudstack-secret is auto-generated based on the k8s cluster owner account, project and env details. I don't think APIs provide an option to provision a custom cloudstack-secret to a custom file.
I guess you can your custom secret by running the following on your control node,

/opt/bin/kubectl -n kube-system create secret generic cloudstack-secret --from-file=<FILE_PATH>

Appropriate provisioning must be done at CloudStack end for API/secret key used in your file.

PS: I've not tested this

[weizhouapache]

+1 with @shwstppr said

The cloudstack-secret is auto-generated by cloudstack, which is important for k8s autoscaler.
If you modify the secret, k8s autoscaler might not work.
https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/cloudstack/README.md

You can create your own secrets in the CKS controller vms.

[ahmadamirahmadi007]

i know cloudstack automatically generated it. cloudstack create a user with the name USER-kubeadmin and used its api-key and secret. this information in visible in K8s. i want change to my own. and i have another question that what is the power of this api url whis corresponding key and can i limit it? @weizhouapache @shwstppr

[shwstppr]

@ahmadamirahmadi007 You'll have to check but the kubeadmin user is created within the k8s cluster owner account. So, API access should be limited to the role of that account.

[weizhouapache]

i know cloudstack automatically generated it. cloudstack create a user with the name USER-kubeadmin and used its api-key and secret. this information in visible in K8s. i want change to my own. and i have another question that what is the power of this api url whis corresponding key and can i limit it? @weizhouapache @shwstppr

@ahmadamirahmadi007

the secret is useful in two scenarios

• LB auto-configuration. If you deploy a k8s service with type as LoadBalancer, ACS will acquire a public IP, and create LB rules automatically.
• autoscaler, K8S will send request to ACS to create new vm or destroy a vm based on the metrics. This can be disabled per CKS cluster.

The corresponding user in the secret must have the privilege to perform the APIs above. Therefore ACS creates the user under the same account as the owner of the kubernetes cluster or the caller (if k8s cluster is created for a project). The privilege depends on the type or role of the account (owner or caller).

If you use the api key of a different user, the above actions might fail.

[ahmadamirahmadi007]

does follow error for this issue?
2023-11-29 18:02:54,639 WARN [c.c.k.c.u.KubernetesClusterUtil] (API-Job-Executor-100:ctx-954247c9 job-21187 ctx-066fcbbe) (logid:17ddb61d) API endpoint for Kubernetes cluster : USERRRRRRRRR not available
javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
Caused by: java.io.EOFException: SSL peer shut down incorrectly
it means k8s can not connect to cloudstack right? @weizhouapache

[weizhouapache]

does follow error for this issue? 2023-11-29 18:02:54,639 WARN [c.c.k.c.u.KubernetesClusterUtil] (API-Job-Executor-100:ctx-954247c9 job-21187 ctx-066fcbbe) (logid:17ddb61d) API endpoint for Kubernetes cluster : USERRRRRRRRR not available javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake Caused by: java.io.EOFException: SSL peer shut down incorrectly it means k8s can not connect to cloudstack right? @weizhouapache

@ahmadamirahmadi007
I think there might be some issues during the deploy/start of k8s API server.
you can ssh into the k8s master nodes and check the k8s logs.

[ahmadamirahmadi007]

I noticed another problem. I don't know if I should ask this question here or in a new topic, but I noticed that if my public address is not behind a proxy, my weave cni does not work at all and my cluster does not start. Is there a way instead of using default repository of weave cni that set in kubernetes iso image. i have a private repository but
Even by setting it in the cluster creation stage, this problem is not solved.

[weizhouapache]

I noticed another problem. I don't know if I should ask this question here or in a new topic, but I noticed that if my public address is not behind a proxy, my weave cni does not work at all and my cluster does not start. Is there a way instead of using default repository of weave cni that set in kubernetes iso image. i have a private repository but
Even by setting it in the cluster creation stage, this problem is not solved.

I have noticed the same problem yesterday.
in control vm and worker vms, vms always try to pull the "latest" images for weave (-kube, -npc, etc) from docker.io repository. However, there are already images (with version 2.8.1) in the CKS ISO.
It is unexpected behavior. It might have caused some smoke test failures with cks due to the rate limitations of dockerhub (see https://docs.docker.com/docker-hub/download-rate-limit/).
I suspect some cks issues are caused by it as well.
I will have a look at the issue next week.
cc @DaanHoogland @shwstppr

update: the network.yaml (download from weavenet) has the following setting

              image: 'weaveworks/weave-kube:latest'
              imagePullPolicy: Always

              image: 'weaveworks/weave-kube:latest'
              imagePullPolicy: Always

              image: 'weaveworks/weave-npc:latest'
              imagePullPolicy: Always

we need to fix it.

[ahmadamirahmadi007]

i used my private registry and create weave yaml manifest with my registry and solved.

[weizhouapache]

i used my private registry and create weave yaml manifest with my registry and solved.

@ahmadamirahmadi007
you just need to use imagePullPolicy: IfNotPresent instead of imagePullPolicy: Always in all the yaml files.
I will create a PR for it and also build some CKS ISOes for testing

[ahmadamirahmadi007]

tnx all

[weizhouapache]

@ahmadamirahmadi007
I have created a PR #8296
and uploaded some ISOes for testing: https://download.cloudstack.org/testing/cks/