Request to create a registries.yaml file as part of k3s installation for private registries

[erikcw]

Is it possible to pass extra k3s configuration to k3sup during setup? For example, I need to add a Private Registry -- which means I need to place /etc/rancher/k3s/registries.yaml on each of my nodes.

Is this something k3sup can do? Or do I need to use ansible or TF to provide not standard config?

[onedr0p]

This tool is really meant for learning and quick testing of a k3s cluster. Once you get passed it's basic use cases I would use a more mature tool like Ansible to install and manage k3s for anything more like production use cases.

Check out this Ansible role:

https://github.com/PyratLabs/ansible-role-k3s

[alexellis]

Hi @onedr0p

Do you think it would be fair to allow the maintainer and/or contributors of k3sup to say what it is intended for?

There are quite a lot of different configuration options, including a HA cluster with a database or etcd, so it's absolutely production ready and can be used with Terraform too.

@erikcw to your point, writing an extra file like registries.yaml could be done via Terraform using a remote SSH provisioner.

Here's a full example of a production ready setup on Linode with a load balancer and Ingres with TLS:

https://www.openfaas.com/blog/production-faas-linode/

It's just a simple text file that could be dropped in at any time correct?

Alex

[alexellis]

I'll get this closed for now and recommend anyone landing here use their Terraform or user-data script to write the registries.yaml file as per:

Upon startup, K3s will check to see if a registries.yaml file exists at /etc/rancher/k3s/ and instruct containerd to use any registries defined in the file. If you wish to use a private registry, then you will need to create this file as root on each node that will be using the registry.

[onedr0p]

Do you think it would be fair to allow the maintainer and/or contributors of k3sup to say what it is intended for?

That's fair, this issue was open for 20 days without a comment so I wanted to offer some guidance with my experience with k3sup and when to use it over more mature options (Terraform/Ansible) for production clusters.

There are quite a lot of different configuration options, including a HA cluster with a database or etcd, so it's absolutely production ready and can be used with Terraform too.

Terraform is not a tool for configuring an operating system after it's installed. Chef, Ansible and Puppet are. Terraform with remote-exec or local-exec is just a major pain in the ass to manage as I am sure you are aware of if you have used provisioners in the past. Even the terraform docs say to use provisioners as a last resort due to "certain behaviors that cannot be directly represented in Terraform's declarative model" and many other important issues laid out in the link.

I'll get this closed for now and recommend anyone landing here use their Terraform or user-data script to write the registries.yaml file as per...

This feature request is out of scope for k3sup? I would hope that k3sup would be able to turn most of the knobs needed for standing up a k3s cluster given it's name but looks like I am wrong here. I would love to replace Ansible with k3sup if I could turn these knobs with k3sup because I don't need the majority of Ansible features for just standing up k3s with some custom configuration.

[alexellis]

That's fair, this issue was open for 20 days without a comment so I wanted to offer some guidance with my experience with k3sup and when to use it over more mature options (Terraform/Ansible) for production clusters.

It's not fair to dump on other projects, when you yourself don't represent them.

Issues can be open for hours, days or weeks before they get commented on. I'm not paid for maintaining, supporting or adding features to K3sup.

Let's set some healthy expectations here. Requests outside of the scope, which have obvious solutions are unlikely to get a response within 1 hour.

Terraform is not a tool for configuring an operating system after it's installed.

Correct, however in this case, the registry configuration is static and part of the total install which involves dedicating a node to work as a K3s worker.

This feature request is out of scope for k3sup? I would hope that k3sup would be able to turn most of the knobs needed for standing up a k3s cluster given it's name but looks like I am wrong here.

If you need terraform and a private registry then that is a commercial/enterprise use-case, OpenFaaS Ltd would be happy to put salaried employees on the features, just reach out to us for an estimate on the costs of the work.