[QUESTION] Auto deploying manifests?

[patoarvizu]

Is there a configurable way to auto-deploy manifests, similar to what k3s allows you to do by placing manifests in /var/lib/rancher/k3s/server/manifests (https://rancher.com/docs/k3s/latest/en/configuration/#auto-deploying-manifests)? Or would the only way be to deploy a custom k3s image with the manifests baked in?

For context, the three use cases I have right now are:

1. I'd like to make sure a StorageClass (like local-path-provisioner) is always bootstrapped.
2. I'd like to customize the Traefik options, specifically to set insecureSkipVerify by default. Or even better, to customize the values on the Traefik HelmChart CRD that is installed by default.
3. I have a service that requires a Secret to exist.

Right now, every time I create a k3d cluster, I have to run 5 or 6 commands to get all of the above to work.

[iwilltry42]

Hey there, thanks for opening this issue 👍

Since k3d is just k3s in docker, you can use the --volume flag to mount your manifests into the exact same directory you mentioned inside the k3d nodes.

E.g. k3d create --volume /path/to/my/manifest:/var/lib/rancher/k3s/server/manifests

Let me know, if that works for you :)

[patoarvizu]

Ah alright, yeah I guess that works too. I'll give it a try and let you know.

It's slightly less than ideal though because I would prefer something that was as simple as (or close to what) k3d c is, but this is good enough.

If I create my custom k3s image with the manifests pre-baked in that location, that should work as well, right? I think using k3d c --image ... might be easier, and is also more repeatable than mounting a volume with manifests that may have changed.

[iwilltry42]

Baking the manifests into the image is definitely the most repeatable way to achieve this.
However, mounting the volume with manifests is more flexible and works just the same everytime as long as you don't modify the files on disk or inside the container.

The only alternative I can think of would be implementing something similar to docker cp for the files. But in the end that'd be the same result.

Or applying the files from a network location.

[patoarvizu]

So, using --volume works but with a caveat, (actually kind of two caveats). Specifically with Traefik, if I put a HelmChart manifest for Traefik in a local directory and mount that with --volume, it will be overwritten by what k3d/k3s create, so it reverts back to the default version of that manifest. What I had to do to get that to work is run k3d c with --server-arg "--no-deploy=traefik" so only my mounted manifest gets deployed. The other semi-caveat is that because the volume is writable in both directions, the other default manifests (coredns.yaml and rolebindings.yaml) are also written to my local volume. That's less of a deal for me, but may be an issue for someone else.

I'll give the custom image a shot soon and can report back soon, but feel free to close the issue too.

Thanks!

[iwilltry42]

So the first caveat is not solvable in the k3d scope and probably that would also not make too much sense, since you can use the --no-deploy=traefik flag for k3s.
We also cannot mount with :ro, because then k3s would fail to write the manifests and error out.
However, I think this could be fixed, if k3s would recursively scan directories under the /var/.../manifests folder for yamls. This way, you can simply mount a folder with manifests into there without conflicting with k3s managed manifests. I'm already playing around with k3s code for this 👍

[iwilltry42]

@patoarvizu I just built iwilltry42/k3s:iwilltry42-master-2019.10.11-1 with my changes included.
Try using this and mount your directory containing the manifests you want to deploy into any subdirectory of /var/lib/rancher/k3s/server/manifests.
E.g. I did k3d create -n test -w 2 --volume /tmp/test-manifests:/var/lib/rancher/k3s/server/manifests/test and it deployed everything that I had on my local machine as well as what k3s creates 👍

[patoarvizu]

Works perfectly, thanks! 👍

[iwilltry42]

k3s-io/k3s#892 just got merged, so we can expect this to be included in the official rancher/k3s:v0.10.0 image 👍