[enterprise-4.16] OSDOCS-10463: vsphere multiple subnets

installing/installing_vsphere/ipi/installing-vsphere-installer-provisioned-network-customizations.adoc

@@ -60,11 +60,24 @@ include::modules/ipi-install-modifying-install-config-for-dual-stack-network.ado
 
 include::modules/configuring-vsphere-regions-zones.adoc[leveloffset=+2]
 
-// begin network customization
+// Network configuration phases
 include::modules/nw-network-config.adoc[leveloffset=+1]
+
+// Specifying advanced network configuration
 include::modules/nw-modifying-operator-install-config.adoc[leveloffset=+1]
+
+// Specifying multiple subnets for your network
+include::modules/nw-operator-vsphere-multiple-subnets.adoc[leveloffset=+2]
+
+[role="_additional-resources"]
+.Additional resources
+
+* xref:../../../installing/installing_vsphere/ipi/installing-vsphere-installer-provisioned-network-customizations.adoc#nw-operator-cr_installing-vsphere-installer-provisioned-network-customizations[Cluster Network Operator configuration]
+
+* xref:../../../rest_api/config_apis/infrastructure-config-openshift-io-v1.adoc#spec-platformspec-vsphere-nodenetworking[`.spec.platformSpec.vsphere.nodeNetworking`]
+
+// Cluster Network Operator configuration
 include::modules/nw-operator-cr.adoc[leveloffset=+1]
-// end network customization
 
 // Services for a user-managed load balancer
 include::modules/nw-osp-services-external-load-balancer.adoc[leveloffset=+1]

installing/installing_vsphere/upi/installing-vsphere-network-customizations.adoc

@@ -61,9 +61,23 @@ include::modules/installation-configure-proxy.adoc[leveloffset=+2]
 
 include::modules/configuring-vsphere-regions-zones.adoc[leveloffset=+2]
 
-// Network Operator specific configuration
+// Network configuration phases
 include::modules/nw-network-config.adoc[leveloffset=+1]
+
+// Specifying advanced network configuration
 include::modules/nw-modifying-operator-install-config.adoc[leveloffset=+1]
+
+// Specifying multiple subnets for your network
+include::modules/nw-operator-vsphere-multiple-subnets.adoc[leveloffset=+2]
+
+[role="_additional-resources"]
+.Additional resources
+
+* xref:../../../installing/installing_vsphere/upi/installing-vsphere-network-customizations.adoc#nw-operator-cr_installing-vsphere-network-customizations[Cluster Network Operator configuration]
+
+* xref:../../../rest_api/config_apis/infrastructure-config-openshift-io-v1.adoc#spec-platformspec-vsphere-nodenetworking[`.spec.platformSpec.vsphere.nodeNetworking`]
+
+// Cluster Network Operator configuration
 include::modules/nw-operator-cr.adoc[leveloffset=+1]
 
 include::modules/installation-generate-ignition-configs.adoc[leveloffset=+1]

modules/nw-network-config.adoc

@@ -25,18 +25,18 @@ Phase 1:: You can customize the following network-related fields in the `install
 * `networking.serviceNetwork`
 * `networking.machineNetwork`
 +
-For more information on these fields, refer to _Installation configuration parameters_.
+For more information, see "Installation configuration parameters".
 +
 [NOTE]
 ====
-Set the `networking.machineNetwork` to match the CIDR that the preferred NIC resides in.
+Set the `networking.machineNetwork` to match the Classless Inter-Domain Routing (CIDR) where the preferred subnet is located.
 ====
 +
 [IMPORTANT]
 ====
-The CIDR range `172.17.0.0/16` is reserved by libVirt. You cannot use this range or any range that overlaps with this range for any networks in your cluster.
+The CIDR range `172.17.0.0/16` is reserved by `libVirt`. You cannot use any other CIDR range that overlaps with the `172.17.0.0/16` CIDR range for networks in your cluster.
 ====
 
 Phase 2:: After creating the manifest files by running `openshift-install create manifests`, you can define a customized Cluster Network Operator manifest with only the fields you want to modify. You can use the manifest to specify advanced network configuration.
 
-You cannot override the values specified in phase 1 in the `install-config.yaml` file during phase 2. However, you can further customize the network plugin during phase 2.
+During phase 2, you cannot override the values that you specified in phase 1 in the `install-config.yaml` file. However, you can customize the network plugin during phase 2.

modules/nw-operator-vsphere-multiple-subnets.adoc

@@ -0,0 +1,68 @@
+// Module included in the following assemblies:
+//
+// * installing/installing_vsphere/ipi/installing-vsphere-installer-provisioned-network-customizations.adoc
+// * installing/installing_vsphere/upi/installing-vsphere-network-customizations.adoc
+
+:_mod-docs-content-type: PROCEDURE
+[id="nw-operator-vsphere-multiple-subnets_{context}"]
+= Specifying multiple subnets for your network
+
+Before you install an {product-title} cluster on a {vmw-short} host, you can specify multiple subnets for a networking implementation so that the {vmw-short} cloud controller manager (CCM) can select the appropriate subnet for a given networking situation. {vmw-short} can use the subnet for managing pods and services on your cluster. 
+
+For this configuration, you must specify internal and external Classless Inter-Domain Routing (CIDR) implementations in the {vmw-short} CCM configuration. Each CIDR implementation lists an IP address range that the CCM uses to decide what subnets interact with traffic from internal and external networks.
+
+[IMPORTANT]
+====
+Failure to configure internal and external CIDR implementations in the {vmw-short} CCM configuration can cause the {vmw-short} CCM to select the wrong subnet. This situation causes the following error:
+
+----
+ERROR Bootstrap failed to complete: timed out waiting for the condition
+ERROR Failed to wait for bootstrapping to complete. This error usually happens when there is a problem with control plane hosts that prevents the control plane operators from creating the control plane.
+----
+
+This configuration can cause new nodes that associate with a `MachineSet` object with a single subnet to become unusable as each new node receives the `node.cloudprovider.kubernetes.io/uninitialized` taint. These situations can cause communication issues with the Kubernetes API server that can cause installation of the cluster to fail.
+====
+
+.Prerequisites
+
+* You created Kubernetes manifest files for your {product-title} cluster.
+
+.Procedure
+
+. From the directory where you store your {product-title} cluster manifest files, open the `manifests/cluster-infrastructure-02-config.yml` manifest file.
+
+. Add a `nodeNetworking` object to the file and specify internal and external network subnet CIDR implementations for the object.
++
+[TIP]
+====
+For most networking situations, consider setting the standard multiple-subnet configuration. This configuration requires that you set the same IP address ranges in the `nodeNetworking.internal.networkSubnetCidr` and `nodeNetworking.external.networkSubnetCidr` parameters.
+====
++
+.Example of a configured `cluster-infrastructure-02-config.yml` manifest file
+[source,yaml]
+----
+apiVersion: config.openshift.io/v1
+kind: Infrastructure
+metadata:
+  name: cluster
+spec:
+  cloudConfig:
+    key: config
+    name: cloud-provider-config
+  platformSpec:
+    type: VSphere
+    vsphere:
+      failureDomains:
+      - name: generated-failure-domain
+      ...
+       nodeNetworking:
+         external:
+           networkSubnetCidr:
+           - <machine_network_cidr_ipv4>
+           - <machine_network_cidr_ipv6>
+         internal:
+           networkSubnetCidr:
+           - <machine_network_cidr_ipv4>
+           - <machine_network_cidr_ipv6>
+# ...
+----