Questions and discussion regarding a full mesh wireguard setup for k3s on cloud and on premise

[uGiFarukh]

Hello,

I am recently trying to use wireguard along with k3s to create a multi-environment cluster setup for my projects. I came across kilo but finding it a bit difficult to setup. So, the basic network topology I am trying to achieve is as follows:

I have an AWS ec2 instance that is running as a k3s master server with mysql datastore. I have two on-premise networks with lots of VMs and IoT devices that I would like to mesh. Also, I have some droplets in digitalocean and vultr that I would also like to mesh together into the same kubernetes k3s cluster. All the nodes will be workers and only the AWS ec2 will be the master node. On premise network, I have a changing ip address that is behind my router's NAT and Firewall. I have some worker node VMs that I would like to add to the k3s cluster without opening any ports or by creating any firewall rules into my router. I have some drones that I might also want to run as a k3s worker node who will be connected to the cluster through my home network or sometimes from outside my home network through LTE or other internet services. Also, being able to access the k3s cluster network and accessing the pods and services through the same secure tunnel would be great.

1, I want to use Full Mesh implementation so that each and every node inside the k3s cluster can talk to each other. How would I implement such a full mesh network with kilo? If the full mesh is implemented, will the encrypted wireguard packets choose the shortest path to any node that it needs to communicate with? Or, I have to define some routing rules inside kilo somehow

2. Is it possible to change the default port kilo uses to setup the wireguard? From default 51820 to something else? Also, is it possible to change the wireguard interface name that kilo creates? From the default kilo0 to something else? Also, can I define my own network pool and subnets which kilo can use to create the mesh network? And maybe assign static ip addresses from the particular subnet to each nodes?

3. Is it possible to change the encryption system that kilo uses to secure the packets? Is it just default wireguard encryption? Is it possible to modify it somehow?

How would I use kilo to setup a full mesh network like this which fulfils the stated requirements? What type of annotations I should use and should I use the kilo flannel implementation in k3s or the full fledged kilo? Any guidance will be very helpful.

I have recently tried to setup kilo but failed maybe because I was not annotating the nodes properly or something of that sort. Also, I have tried using the k3s wireguard flannel backend but it works properly when all nodes are in the same network. If nodes are in different network then an annotation patching container needs to run to correct flannel external ip addresses. But then again, the metrics server don't work and cannot do api calls to retrieve node resource utilization data and thus kubernetes autoscaling also do not work. I am hoping the kilo implementation will actually be able to access port 10250 on all nodes (even behind NAT/Firewall) which is used by the metrics server to query all nodes for resource metrics.

Thank You

[DeamonLuck]

For Q2. I think it's possible.
The documentation for kg list a bunch of options that you can pass.
I believe --port is the one you want.