New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bug 1838032: Better docs for using custom subnets #3647
Bug 1838032: Better docs for using custom subnets #3647
Conversation
@iamemilio: This pull request references Bugzilla bug 1838032, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker. 3 validation(s) were run on this bug
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
docs/user/openstack/customization.md
Outdated
@@ -137,6 +138,11 @@ platform: | |||
clusterOSImage: my-rhcos | |||
``` | |||
|
|||
## Custom Subnets | |||
|
|||
If you want to install your cluster on a custom network and subnet, you can pass the ID of the subnet to the install config with the `machinesSubnet` option. This subnet will be tagged as the primary subnet for the cluster. The nodes, and VIP port will be created on this subnet. This does add some complexity to the install process that needs to be accounted for. First of all, the CIDR of the custom subnet must match the cidr of `networks.machineNetwork`. Also be aware that the API and Ingress VIPs by default will try to take the .5 and .7 of your network CIDR. If they are not available, then you should set them in the install config by setting the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The user-provided subnet needs to have a DHCP server. This will likely always be the case, but better be explicit about the requirement.
Also, we should explain how to avoid potential IP address conflicts with the API and Ingress VIPs. These IP should be picked outside of the DHCP range and passed to the installer via the apiVIP
and ingressVIP
options. It would actually be better if we could let the DHCP assign the VIPs on the ports.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How would we have dhcp assign the ports? I looked into allowing the ports to be set dynamically in the installer and the way things are right now, it is not doable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using the CLI, by specifying --fixed-ip "subnet=<my-subnet>"
, it's possible to let the DHCP assign an IP to the port:
$ openstack port create --network mandre-krc6g-openshift --security-group mandre-krc6g-worker --fixed-ip "subnet=mandre-krc6g-nodes" --tag openshiftClusterID="mandre-krc6g" "mandre-krc6g-test-port"
$ openstack port show mandre-krc6g-test-port -f value -c fixed_ips
[{'subnet_id': 'c72ac7d7-2f46-4698-92ad-22458089de7d', 'ip_address': '10.0.128.34'}]
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the installer doesn't let you pass a port, it only lets you pass it an IP address to make a port from. I am proposing changing it to allow you to pass the ports directly here though: openshift/enhancements#342
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well, the installer creates the port. It's the same thing. I showed how it worked with the CLI. In terraform, you'd have to pass a fixed_ip
stuct with a subnet_id
for the port creation.
https://www.terraform.io/docs/providers/openstack/r/networking_port_v2.html#fixed_ip
/lgtm |
/hold until I close all the comments |
/hold cancel |
docs/user/openstack/customization.md
Outdated
@@ -110,7 +111,7 @@ To do so and upload binary data from a custom location the user may set `cluster | |||
**NOTE:** The optional `sha256` query parameter can be attached to the URL, which will force the installer to check the image file checksum before uploading it into Glance. | |||
|
|||
Example: | |||
|
|||
`networks.machineNetwork` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems like this addition has nothing to do here :)
docs/user/openstack/customization.md
Outdated
@@ -137,6 +138,10 @@ platform: | |||
clusterOSImage: my-rhcos | |||
``` | |||
|
|||
## Custom Subnets | |||
|
|||
If you want to install your cluster on a custom network and subnet, you can pass the ID of the subnet to the install config with the `machinesSubnet` option. This subnet will be tagged as the primary subnet for the cluster. The nodes, and VIP port will be created on this subnet. This does add some complexity to the install process that needs to be accounted for. First of all, the subnet must have DHCP enabled. Also, the CIDR of the custom subnet must match the cidr of `networks.machineNetwork`. Be aware that the API and Ingress VIPs by default will try to take the .5 and .7 of your network CIDR. If they are not available, then you should set them in the install config by setting the `apiVIP` and `ingressVIP` options to the IP addresses you want to reserve. Lastly, note that setting `externalDNS` while setting `machinesSubnet` is invalid usage, since the installer will not modify your subnet. If you want to add a DNS to your cluster while using a custom subnet, add it to the subnet in openstack [like this](https://docs.openstack.org/neutron/rocky/admin/config-dns-res.html). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd like a comment somewhere saying that apiVIP
and ingressVIP
should be picked outside of the DHCP allocation range in order to avoid any IP address conflicts.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hmm yeah that is wise, I will add that
ebfca77
to
6debf76
Compare
docs/user/openstack/customization.md
Outdated
@@ -137,6 +138,12 @@ platform: | |||
clusterOSImage: my-rhcos | |||
``` | |||
|
|||
## Custom Subnets | |||
|
|||
Before running the installer with a custom `machiensSubnet`, there are a few things you should check. First, make sure that you have a network and subnet available. Note that the installer will create ports on this network, and some ports will be given a fixed IP addresses in the subnet's CIDR. Make sure that the credentials that you give the installer are authorized to do this! The installer will also apply a tage to the subnet. Lastly, the installer will not create a router, so make sure to do this if you need a router to attach floating ip addresses to the network you want to install the nodes onto. Note that for now, this feature does not fully support provider networks. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@iamemilio "machiensSubnet" -> "machinesSubnet"
6debf76
to
f681266
Compare
docs/user/openstack/customization.md
Outdated
@@ -137,6 +138,12 @@ platform: | |||
clusterOSImage: my-rhcos | |||
``` | |||
|
|||
## Custom Subnets | |||
|
|||
Before running the installer with a custom `machinesSubnet`, there are a few things you should check. First, make sure that you have a network and subnet available. Note that the installer will create ports on this network, and some ports will be given a fixed IP addresses in the subnet's CIDR. Make sure that the credentials that you give the installer are authorized to do this! The installer will also apply a tage to the subnet. Lastly, the installer will not create a router, so make sure to do this if you need a router to attach floating ip addresses to the network you want to install the nodes onto. Note that for now, this feature does not fully support provider networks. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/tage/tag/
Do we track the support (or lack of) custom subnets and provider networks somewhere? Not asking to put the link in the doc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not sure, but this has come up a few times as a common misunderstanding. If not here, we should make a note of it somewhere at least
f681266
to
fb30018
Compare
I personally find it a little too verbose. Not a bit deal, it still provides good information. We can rework the docs later if it's really bothering us. |
/lgtm |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Fedosin The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
@iamemilio: Bugzilla bug 1838032 is in an unrecognized state (VERIFIED) and will not be moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
This beefs up the openstack docs for installing with a custom subnet.
/cc @pierreprinetti @maxwelldb @morenod
/label platform/openstack