Using kubernetes driver with porter

[martin-schoenegger]

Hello everyone,

I am trying to run a service using the porter sdk on k3s. There is no docker installed but containerd is.
So with the current setup I can't mount /var/run/docker.sock into the container.

Referring to https://github.com/getporter/porter/blob/main/pkg/cnab/drivers/drivers.go#L21
Is there currently a way to configure the "kubernetes" driver (= --driver kubernetes option) or will this come soon?
I have seen configurations for this in porter/operator
E.g. using environment variables or configuration options via the sdk.

In getporter/porter setting

porter.InstallOptions.Driver="kubernetes"

is not sufficient. It will throw:

Error: 1 Error occurred:
* error retrieving external kubernetes konfiguration for with kubeconfig : invalid configuration: no configuration has been provided

My other option would be to switch to k3s using docker instead of containerd but I wanted to evaluate my options with porter first.

Best regards,

Martin

[carolynvs]

The code in the porter operator repository is a proof of concept so any use bits of code are totally fair game to move into porter directly if we think the porter cli can use them too. So if we wanted to add support for --driver kuberentes, I'd be happy to chat more about how that should look and help someone get a PR merged with the changes.

I am actively working on getting porter altered so that we can support a proper operator. This will mean that porter itself will understand how to load files (that 99% match the operator's CRDs) for installations, credential sets and parameter sets, and then reconcile them. So that may be something to keep in mind as you are working on your service too.

The situation is that the cnab-go library has a kubernetes driver, but it requires a lot of extra configuration to work properly. For example a volume for sharing files between the pod running the bundle and the pod where porter is orchestrating the bundle. The operator is currently taking care of setting all the required environment variables so that they are passed through to the cnab-go library.

An official Porter Kubernetes driver would need to come up with a design for how the end user running porter should set these values (if any, maybe most can be calculated). Some questions you'd want to look at are:

• Is porter guaranteed to be running inside the target cluster? I don't think there is a good solution now to target k8s when porter is running outside of the cluster.
• Is porter responsible for creating the volume shared with the bundle installer? Or is that something the user would specify?
• What service account does the bundle run as?

Let me know if you are interested in working on this (even if it's just the design) or if this is a feature request. 😀

[martin-schoenegger]

Thank you for your response.
This is a feature request and I am willing to contribute.
If you click on my profile you will notice I am quite new to the open source community and I would need some hints.
If that is possible (in whatever way) I am happy to contribute.

Best regards

Martin

[carolynvs]

Cool! So when you are using porter here, are you using porter inside the cluster? I'm trying to work out if we can just grab the current kubernetes config that is available to code running inside the cluster and provide that to the kubeconfig parameter that the bundle is complaining about.

[martin-schoenegger]

Yes it is running in a container on k3os and the kubeconfig file is accessible. The path to kubeconfig could be provided similar as helm does via the KUBECONFIG environment variable or via - - kubeconfig=/path/to/kubeconfig

[carolynvs]

Let's think through what porter install --driver kubernetes would need to do.

1. Make a kubeconfig that represents the in-cluster connection to the current cluster that it is running within. You said that you have a kubeconfig available, but I am not sure that would be the case for all clusters? So we would want to figure out if we can rely on that or make our own.
2. Consider how the driver should "find" parameters and credentials that it can satisfy. Do we just rely on well-known names, such as kubeconfig?
3. Tweak how Porter interacts with its drivers so that drivers can provide parameters and credentials to a bundle. The driver would then give porter the kubeconfig to satisfy the kubeconfig credential defined by the bundle.
4. Take the code from the Porter Operator that handles making a PVC, setting a bunch of extra env vars, etc and turn that into a kubernetes driver inside the getporter/porter repository. What we would probably do is have this driver wrap the kubernetes driver defined in github.com/cnabio/cnab-go. We can reuse the base functionality of that driver (like the Porter Operator does today) but consolidate all the extra logic that Porter needs to configure it properly.