[FEATURE] Allow using existing (or default) docker bridge network, to share network between clusters

[adrienjt]

Hi, I'd like to use k3d as a lightweight solution to test multi-cluster controllers. Currently, there's no easy way to call the Kubernetes API of, e.g., k3d-cluster1 from a pod in k3d-cluster2. I can connect the server container of cluster1 to the cluster2 bridge network and vice versa (with docker metwork connect), but cluster1's server certificate isn't valid for an IP in cluster2.

With kind, clusters are created in the default docker bridge network, so they are reachable from one another.

A --network DOCKER_NETWORK_NAME option could be added to the k3d create subcommand. The network would be created if it doesn't exist.

This issue differs from PR #53 in that it is more general (a --host option could actually also be used to use the host network rather than a bridge network).

[malaDev]

I will investigate about this one, but any insights would be much appreciated

[iwilltry42]

Re-Using existing networks or using the default bridge network sounds like a good idea 👍
I can have a look, if we can easily add a flag here tomorrow. Sorry, I'm quite busy recently and am refactoring k3d completely on the side, so I didn't find the time for this yet...

[malaDev]

the new flag is quite ready but there is some concern about the way to deal with creating/connecting to the chosen network

I noticed that by default, a network with name of the cluster is always created, I guess this should be conserved.

And additionally I will just add or use a network to connect the cluster to. So far it seems to be easy enough but I'm wondering if there is not something else to carry about

[luisdavim]

This is my current workaround, on each cluster I create a service and an endpoint pointing to the other clusters, using a template like this:

DOCKER_GW=$(docker run -it --rm docker.prod.walmart.com/wce/alpine sh -c 'route | awk "/^default/ { print \$2 }"')
cat << EOF > endpoints-${NAME}.yaml
kind: Service
apiVersion: v1
metadata:
  name: ${NAME}
  namespace: default
spec:
  ports:
  - protocol: TCP
    port: ${PORT}
    targetPort: ${PORT}
---
kind: Endpoints
apiVersion: v1
metadata:
  name: ${NAME}
  namespace: default
subsets:
- addresses:
  - ip: ${DOCKER_GW}
  ports:
  - port: ${PORT}
EOF

$NAME and $PORT refer to the name and api port of the cluster I want to connect to

[msaunois-dzr]

Hello

I have a Consul and a Vault instance in containers attached to a specific Docker network (let's say hashicorp_network. If I connect the k3d containers, I can resolve the IP of my services from inside those them:

$ docker exec k3d-dzr-k3s-server ping consul
PING consul (172.19.0.4): 56 data bytes
64 bytes from 172.19.0.4: seq=0 ttl=64 time=0.077 ms
64 bytes from 172.19.0.4: seq=1 ttl=64 time=0.269 ms

But if I ping the same service from inside a pod, the resolution does not work:

$ kubectl run --restart=Never --rm -i --tty tmp --image=alpine -- sh
If you don't see a command prompt, try pressing enter.
/ # ping -w 5 consul
PING consul (10.101.252.23): 56 data bytes

--- consul ping statistics ---
6 packets transmitted, 0 packets received, 100% packet loss

I can reach the containers by their IPs inside the pods though. So it is only a DNS issue. I tried to create a service and a endpoint like @luisdavim but without success. Can someone give me some clues please?

Edit: I'm on Linux.

[iwilltry42]

--network flag is present in k3d v3 (see https://k3d.io/internals/networking/ for more details)