Clarify potential for connection disruption due to --native-routing-cidr flag

[joestringer]

In Cilium 1.8, we intend to change the masquerade logic to more accurately determine when to apply masquerade (ie SNAT) to traffic in direct-routing mode, to ensure that traffic is only masqueraded when the destination is not directly reachable. To do this, we solicit the directly routable CIDR range from the user via the --native-routing-cidr option.

The upgrade docs already describe this briefly:
https://github.com/cilium/cilium/blob/master/Documentation/install/upgrade.rst#important-changes-required-before-upgrading-to-180

By my understanding, if this option is not configured, this will result in behaviour change for direct routing users which may lead to connection disruption. We should briefly extend it to clearly define the consequences of failing to configure the option to ensure that users treat this option with the severity it requires.

Questions to guide this documentation:

• Are all existing direct-routing users already explicitly configuring --ipv4-cluster-cidr-mask-size or is it automatically configured? If it is automatically configured, users may not be aware that they were running with this option.
  • By my initial grep for this option in the Cilium-1.7 codebase, it looks like no instructions inform the user that they must specify this CIDR.
  • If it's autodetected today, then we should make this clear that users who have no knowledge of this setting, but are running --tunnel=disabled will still need to read the paragraph and take action. This can be achieved with a short sentence that states what the default behaviour is, eg "Cilium 1.7 and below inferred the setting for this value if it was not explicitly set."
• What are the consequences of failing to configure this option?
  • For example, cilium-managed pods may be unable to establish connections to the outside world(?), (or to non-Cilium-managed IPs within the cluster?).
• Why does the user need to consider this now?
  • For example, "Previous Cilium versions inferred the cluster range to choose when to perform masquerading, however in some cases it would mistakenly masquerade traffic that is destined outside the cluster".