Running KCP on kubernetes.

[yogesh-reddy]

Hi Team,
I had a use case on running KCP on Kubernetes. Since the helm chart is pretty old i had no other option.. tried docker build and used the image and ran a pod on that image..
And I have created a service that connects to the pod on port 6443 ..
but how can i access the kcp api-server .. i mean how can I get the kubeconfig etc..
for starters I have tried referencing the service kubectl -s <svc-adress> api-resources . but not successful.

I might be doing it in a completely wrong way or there might be a prescribed way of which I am unaware..

Can you please guide through this ...
need this help badly

[hardys]

Hi @yogesh-reddy to help I think we'll need to see your manifests (or yaml for all the created resources if you set this up manually)

It sounds like you're trying to just deploy a single KCP API without any front-proxy? (which should be possible, but is different to the manifests in the kcp repo)

Also more information on how you wish to connect to the API would be useful - e.g creating a Service is enough for connectivity inside the cluster, but you'll also need a Route if you want external connectivity

[yogesh-reddy]

@hardys i did not look at the manifests that you shared previously ... what i did was pure entry-level trial..
Used this https://github.com/kcp-dev/kcp/blob/main/Dockerfile file and tried building a docker image and ran that image as a pod and the service exposed to that pod on 6443 . and i was pretty sure that doesn't work.

So here is my use case ..
lets say i have a reconciler running for some ABC CR in my Kubernetes cluster and lets say it has some deployment related info .. the reconciler takes that info and tries to connect to keep control plane and apply those deployments on kcp control plane through which it gets communicated to other physical clusters..

so to make this. happened my first goal is to run kcp on k8s. hope that makes sense to you.

If I am unable to express this use case can we get on a quick call and discuss it at any time possible for you?

Thank you

[hardys]

Thanks for the information - from the slack discussion it sounds like you would like a minimal example to start with, so I pushed this manifest which shows a super-minimal example starting single-process KCP with healthchecks and a service (though obviously for non-development usage something similar to the manifests previously mentioned would be required).

With that applied we can see KCP is running:

$ kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
kcp-5dd859695c-9z9d5   1/1     Running   0          9m35s

We can access the API using the generated admin kubeconfig (here I'm just accessing from inside the container but you should be able to access from another container via the Service also):

$ kubectl exec -it kcp-5dd859695c-9z9d5 -- env KUBECONFIG=/tmp/admin.kubeconfig kubectl workspace
Current workspace is "kvdk2spgmbix".
Note: 'kubectl ws' now matches 'cd' semantics: go to home workspace. 'kubectl ws -' to go back. 'kubectl ws .' to print current workspace.

To access via the service you may also want to create some additional kubeconfig, and you could use the TLS files generated to create another client cert (or use something like cert-manager as in the manifest examples):

$ kubectl exec -it kcp-5dd859695c-9z9d5 -- ls /tmp
admin.kubeconfig  apiserver.crt     apiserver.key     etcd-server       sa.key

[yogesh-reddy]

@hardys Thanks a ton this works.
but lets say i want a pod running on the same cluster to connect to this KCP pod and perform some operations. since its a communication with the same cluster, i might not need this TLS enabled .. in such a case can I use an unsecured way with out certs or TLS ..

or lets if the above option is not possible is there a way to generate a kubeconfig that can be used to connect to kcp running on this pod .. i can ship that kubeconfig anywhere and can use this.. like use the kubeconfig in my local system and connect to kcp running as pod on my EKS cluster.

[hardys]

@hardys Thanks a ton this works. but lets say i want a pod running on the same cluster to connect to this KCP pod and perform some operations. since its a communication with the same cluster, i might not need this TLS enabled .. in such a case can I use an unsecured way with out certs or TLS ..

KCP is built on the kube apiserver code, which no longer supports running without TLS, therefore you can't run KCP without TLS

or lets if the above option is not possible is there a way to generate a kubeconfig that can be used to connect to kcp running on this pod .. i can ship that kubeconfig anywhere and can use this.. like use the kubeconfig in my local system and connect to kcp running as pod on my EKS cluster.

There are several ways to make a service accessible outside a cluster, which one you choose will depend on your environment and what style of Ingress it supports, but if you check the helm templates there are some examples which could be adapted.

You will also need to choose the authentication method - again there are several options supported by KCP, and the flags are the same as kube apiserver so you can refer to the upstream docs for more details

[yogesh-reddy]

Also there is no way here for an onboarded workload cluster to communicate with kcp server right? like how does syncer pod on my physical cluster know how to connect to kcp running as a pod in some cluster?

[hardys]

Also there is no way here for an onboarded workload cluster to communicate with kcp server right? like how does syncer pod on my physical cluster know how to connect to kcp running as a pod in some cluster?

Check the docs for more information on how to configure the syncer - it uses a serviceaccount token to communicate with KCP.

[hardys]

@yogesh-reddy I don't think there are any KCP specific issues here, shall we close this issue and track work to improve the helm charts in that repo?

[yogesh-reddy]

Sure @hardys .. will close this and thanks for input.