Unable to create second cluster

[drewwells]

What happened:
When using a config, it's not possible to create a second cluster. It times out waiting for kubelet to become ready. Works fine if you remove the --config parameter. I create clusters at the same time, so logs may say cluster1 failed but it's because it created after cluster2
cluster2.tar.gz
What you expected to happen:
Two kind clusters come up

How to reproduce it (as minimally and precisely as possible):

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker

kind create cluster --name cluster1 --config config.yaml
kind create cluster --name cluster2 --config config.yaml

Anything else we need to know?:
It works if you take the config off and just create vanilla kind clusters
Repo'd in kind 0.18.0 and 0.20.0
Environment:

• kind version: (use kind version): 0.20.0
• Runtime info: (use docker info or podman info):
• OS (e.g. from /etc/os-release): linux
• Kubernetes version: (use kubectl version): n/a
• Any proxies or other special environment settings?:

[stmcginnis]

The difference between with the config and without is you are doubling the number of containers being created. It's quite possible this is pushing you into a resource issue. One common issue is with inotify, but it can also be an issue with CPU or memory.

Please fill in the Environment questions to help understand what your environment looks like, but I would recommend taking a look at some of the resource consumption when you try to create this many nodes/containers.

[stmcginnis]

Here's another recent, potentially relevant possibility: #2744 (comment)

[adelton]

I used the approach in #2744 (comment) to allow me to create two clusters (each with one control-plane and two workers), on a Fedora 38 with rootless podman. Otherwise the second cluster got stuck at the Joining worker nodes step.

[drewwells]

I used the approach in #2744 (comment) to allow me to create two clusters (each with one control-plane and two workers), on a Fedora 38 with rootless podman. Otherwise the second cluster got stuck at the Joining worker nodes step.

Testing this, thanks.

The server is very beefy. I also didn't see resource limits being hit when using -v8 on the create command

[drewwells]

That fixed the issue. I don't know how I was going to find that based on the errors encountered. It would be helpful perhaps to have a page on setting up multiple kind clusters on one box... It also doesn't explain why kind create works but kind create --config {default config} doesn't.

[BenTheElder]

https://kind.sigs.k8s.io/docs/user/known-issues/

YMMV, my own hosts with no configuration changes can create multiple clusters fine, it depends on both the hosts and the other workloads colocated on that host. E.G. your IDE may consume too many inotify watches versus your system config.

[BenTheElder]

This is a general linux issue that is surfaced by kind because it shares your host kernel via a container rather than say, a VM based node solution, and kubernetes + kubernetes workloads can use a lot of resources.