Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug 1838032: Better docs for using custom subnets #3647

Merged

Conversation

iamemilio
Copy link

This beefs up the openstack docs for installing with a custom subnet.

/cc @pierreprinetti @maxwelldb @morenod
/label platform/openstack

@iamemilio iamemilio changed the title Better docs for using custom subnets Bug 1838032: Better docs for using custom subnets May 26, 2020
@openshift-ci-robot openshift-ci-robot added the bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. label May 26, 2020
@openshift-ci-robot
Copy link
Contributor

@iamemilio: This pull request references Bugzilla bug 1838032, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.5.0) matches configured target release for branch (4.5.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1838032: Better docs for using custom subnets

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label May 26, 2020
@@ -137,6 +138,11 @@ platform:
clusterOSImage: my-rhcos
```

## Custom Subnets

If you want to install your cluster on a custom network and subnet, you can pass the ID of the subnet to the install config with the `machinesSubnet` option. This subnet will be tagged as the primary subnet for the cluster. The nodes, and VIP port will be created on this subnet. This does add some complexity to the install process that needs to be accounted for. First of all, the CIDR of the custom subnet must match the cidr of `networks.machineNetwork`. Also be aware that the API and Ingress VIPs by default will try to take the .5 and .7 of your network CIDR. If they are not available, then you should set them in the install config by setting the
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The user-provided subnet needs to have a DHCP server. This will likely always be the case, but better be explicit about the requirement.

Also, we should explain how to avoid potential IP address conflicts with the API and Ingress VIPs. These IP should be picked outside of the DHCP range and passed to the installer via the apiVIP and ingressVIP options. It would actually be better if we could let the DHCP assign the VIPs on the ports.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How would we have dhcp assign the ports? I looked into allowing the ports to be set dynamically in the installer and the way things are right now, it is not doable.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using the CLI, by specifying --fixed-ip "subnet=<my-subnet>", it's possible to let the DHCP assign an IP to the port:

$ openstack port create --network mandre-krc6g-openshift --security-group mandre-krc6g-worker --fixed-ip "subnet=mandre-krc6g-nodes" --tag openshiftClusterID="mandre-krc6g" "mandre-krc6g-test-port"
$ openstack port show mandre-krc6g-test-port -f value -c fixed_ips                                                                                                                                       
[{'subnet_id': 'c72ac7d7-2f46-4698-92ad-22458089de7d', 'ip_address': '10.0.128.34'}]

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the installer doesn't let you pass a port, it only lets you pass it an IP address to make a port from. I am proposing changing it to allow you to pass the ports directly here though: openshift/enhancements#342

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, the installer creates the port. It's the same thing. I showed how it worked with the CLI. In terraform, you'd have to pass a fixed_ip stuct with a subnet_id for the port creation.

https://www.terraform.io/docs/providers/openstack/r/networking_port_v2.html#fixed_ip

@Fedosin
Copy link
Contributor

Fedosin commented May 27, 2020

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label May 27, 2020
@iamemilio
Copy link
Author

/hold until I close all the comments

@openshift-ci-robot openshift-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. and removed lgtm Indicates that a PR is ready to be merged. labels May 27, 2020
@iamemilio
Copy link
Author

/hold cancel

@openshift-ci-robot openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 27, 2020
@@ -110,7 +111,7 @@ To do so and upload binary data from a custom location the user may set `cluster
**NOTE:** The optional `sha256` query parameter can be attached to the URL, which will force the installer to check the image file checksum before uploading it into Glance.

Example:

`networks.machineNetwork`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like this addition has nothing to do here :)

@@ -137,6 +138,10 @@ platform:
clusterOSImage: my-rhcos
```

## Custom Subnets

If you want to install your cluster on a custom network and subnet, you can pass the ID of the subnet to the install config with the `machinesSubnet` option. This subnet will be tagged as the primary subnet for the cluster. The nodes, and VIP port will be created on this subnet. This does add some complexity to the install process that needs to be accounted for. First of all, the subnet must have DHCP enabled. Also, the CIDR of the custom subnet must match the cidr of `networks.machineNetwork`. Be aware that the API and Ingress VIPs by default will try to take the .5 and .7 of your network CIDR. If they are not available, then you should set them in the install config by setting the `apiVIP` and `ingressVIP` options to the IP addresses you want to reserve. Lastly, note that setting `externalDNS` while setting `machinesSubnet` is invalid usage, since the installer will not modify your subnet. If you want to add a DNS to your cluster while using a custom subnet, add it to the subnet in openstack [like this](https://docs.openstack.org/neutron/rocky/admin/config-dns-res.html).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like a comment somewhere saying that apiVIP and ingressVIP should be picked outside of the DHCP allocation range in order to avoid any IP address conflicts.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm yeah that is wise, I will add that

@iamemilio iamemilio force-pushed the Enhance_BYO_docs_openstack branch 2 times, most recently from ebfca77 to 6debf76 Compare May 29, 2020 20:13
@@ -137,6 +138,12 @@ platform:
clusterOSImage: my-rhcos
```

## Custom Subnets

Before running the installer with a custom `machiensSubnet`, there are a few things you should check. First, make sure that you have a network and subnet available. Note that the installer will create ports on this network, and some ports will be given a fixed IP addresses in the subnet's CIDR. Make sure that the credentials that you give the installer are authorized to do this! The installer will also apply a tage to the subnet. Lastly, the installer will not create a router, so make sure to do this if you need a router to attach floating ip addresses to the network you want to install the nodes onto. Note that for now, this feature does not fully support provider networks.
Copy link
Contributor

@maxwelldb maxwelldb May 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@iamemilio "machiensSubnet" -> "machinesSubnet"

@@ -137,6 +138,12 @@ platform:
clusterOSImage: my-rhcos
```

## Custom Subnets

Before running the installer with a custom `machinesSubnet`, there are a few things you should check. First, make sure that you have a network and subnet available. Note that the installer will create ports on this network, and some ports will be given a fixed IP addresses in the subnet's CIDR. Make sure that the credentials that you give the installer are authorized to do this! The installer will also apply a tage to the subnet. Lastly, the installer will not create a router, so make sure to do this if you need a router to attach floating ip addresses to the network you want to install the nodes onto. Note that for now, this feature does not fully support provider networks.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/tage/tag/
Do we track the support (or lack of) custom subnets and provider networks somewhere? Not asking to put the link in the doc.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure, but this has come up a few times as a common misunderstanding. If not here, we should make a note of it somewhere at least

@mandre
Copy link
Member

mandre commented Jun 5, 2020

I personally find it a little too verbose. Not a bit deal, it still provides good information. We can rework the docs later if it's really bothering us.
/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jun 5, 2020
@adduarte
Copy link

adduarte commented Jun 5, 2020

/lgtm

@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Fedosin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 6, 2020
@openshift-merge-robot openshift-merge-robot merged commit f206ff1 into openshift:master Jun 6, 2020
@openshift-ci-robot
Copy link
Contributor

@iamemilio: Bugzilla bug 1838032 is in an unrecognized state (VERIFIED) and will not be moved to the MODIFIED state.

In response to this:

Bug 1838032: Better docs for using custom subnets

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged. platform/openstack
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants