bootstrap fails with Invalid networks:service CIDR - no route exists for 10.96.0.0/12

[brembelguebli]

Component:

bootstrap

What happened:
bootstrap fails with
Invalid networks:service CIDR - no route exists for 10.96.0.0/12

What was expected:
Success

Steps to reproduce
Follow the documentation at https://documentation-internal.scality.com/RINGX/0.3.0/MetalK8s/installation/bootstrap.html, define the networks pods and services , run bootstrap from mk8s-2.8

Resolution proposal (optional):

Succeed may be ?

[NicolasT]

Well, is there a route for 10.96.0.0/12? What does ip r get 10.96.0.1 say? This is a pre-check, we need such route (or a default route which spans said subnet, or whatever) to exist, otherwise the cluster won't work. Looks like the pre-check is failing.

If no default route can be defined, then you can create a dummy interface and assign a route to 10.96.0.0/12 to that dev.

[brembelguebli]

I am at the bootstrap phase so far, so nothing has been deployed, the services network is supposed to be an internal network for mk8s where the services (DNS, etc...) are hosted..
chiken and egg dilemma ?

[NicolasT]

The ClusterIP 'network' is not really a network, it's a range of addresses which will be assigned to Service objects of type ClusterIP in K8s. Traffic to such ClusterIP addresses is managed by kube-proxy which configures a bunch of iptables rules to get this done.

Now, in Linux, when a connection is made to an address and there's no entry in the routing table matching said address, the connection attempt is refused, and doesn't even end up in the iptables layer.

So, MetalK8s (and K8s in general) requires the ClusterIP address range to be covered by a route on the system. In general, having a default route covers that nicely, nothing to be done. In environments where no default route exists, some route needs to be set up to cover the ClusterIP range, even if there's no physical network backing this network range (and indeed, the address range should not be used for real!).

Hence, using a 'dummy' link and a route to said dev fulfills the requirements.

To summarize: we don't need the network to exist (to the contrary). We do need a route for the network to exist (even if that route goes nowhere at all, that's fine, it's not really used anyway).

[NicolasT]

@scality/metalk8s I couldn't immediately find something in the docs explaining what causes this pre-check to fail and what can be done about it. May want to improve on this.

[brembelguebli]

indeed no default route on the server
doing a route add -net default dev ens1f0 makes things go further indeed

[thomasdanan]

I guess we should add a note about this in the troubleshooting section: https://metal-k8s.readthedocs.io/en/2.8.0/installation/troubleshooting/bootstrap_installation_errors.html

[NicolasT]

I guess we should add a note about this in the troubleshooting section: https://metal-k8s.readthedocs.io/en/2.8.0/installation/troubleshooting/bootstrap_installation_errors.html

Indeed. Let's track that in another ticket.

Also, going to move this one in Discussions.

[thomasdanan]

By the way, maybe another way or complementary way to help the user in such a situation is to have the troubleshooting note displayed in the console when the pre-check failed (so that the user doesn't even have to look for the needed information in the documentation):

Invalid networks:service CIDR - no route exists for 10.96.0.0/12 - please, create the route using the following command: xxxxx or check the documentation here