Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
The experimental OpenStack backend used to create an extra server running DNS and load balancer services that the cluster needed. OpenStack does not always come with DNSaaS or LBaaS so we had to provide the functionality the OpenShift cluster depends on (e.g. the etcd SRV records, the api-int records & load balancing, etc.). This approach is undesirable for two reasons: first, it adds an extra node that the other IPI platforms do not need. Second, this node is a single point of failure. The Baremetal platform has faced the same issues and they have solved them with a few virtual IP addresses managed by keepalived in combination with coredns static pod running on every node using the mDNS protocol to update records as new nodes are added or removed and a similar static pod haproxy to load balance the control plane internally. The VIPs are defined here in the installer and they use the PlatformStatus field to be passed to the necessary machine-config-operator fields: openshift/api#374 The Bare Metal IPI Networking Infrastructure document is broadly applicable here as well: https://github.com/openshift/installer/blob/master/docs/design/baremetal/networking-infrastructure.md Notable differences in OpenStack: * We only use the API and DNS VIPs right now * Instead of Baremetal's Ingress VIP (which is attached to the OpenShift routers) our haproxy static pods balance the 80 & 443 pods to the worker nodes * We do not run coredns on the bootstrap node. Instead, bootstrap itself uses one of the masters for DNS. These differences are not fundamental to OpenStack and we will be looking at aligning more closely with the Baremetal provider in the future. There is also a great oportunity to share some of the configuration files and scripts here. This change needs several other pull requests: Keepalived plus the coredns & haproxy static pods in the MCO: openshift/machine-config-operator/pull/740 Passing the API and DNS VIPs through the installer: #1998 Vendoring the OpenStack PlatformStatus changes in the MCO: openshift/machine-config-operator#978 Allowing to use PlatformStatus in the MCO templates: openshift/machine-config-operator#943 Co-authored-by: Emilio Garcia <egarcia@redhat.com> Co-authored-by: John Trowbridge <trown@redhat.com> Co-authored-by: Martin Andre <m.andre@redhat.com> Co-authored-by: Tomas Sedovic <tsedovic@redhat.com> Massive thanks to the Bare Metal and oVirt people!
- Loading branch information
1 parent
5a56a7e
commit 53e3bf3
Showing
22 changed files
with
277 additions
and
584 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
25 changes: 25 additions & 0 deletions
25
data/data/bootstrap/openstack/files/etc/keepalived/keepalived.conf.tmpl
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
vrrp_script chk_ocp { | ||
# NOTE(mandre) the fake kube-api server doesn't responds to the | ||
# https://0:6443/readyz URL, we need to find another check | ||
script "ss -tnl | grep 6443" | ||
interval 1 | ||
weight 50 | ||
} | ||
|
||
vrrp_instance ${CLUSTER_NAME}_API { | ||
state BACKUP | ||
interface ${INTERFACE} | ||
virtual_router_id ${API_VRID} | ||
priority 50 | ||
advert_int 1 | ||
authentication { | ||
auth_type PASS | ||
auth_pass ${CLUSTER_NAME}_api_vip | ||
} | ||
virtual_ipaddress { | ||
${API_VIP}/${NET_MASK} | ||
} | ||
track_script { | ||
chk_ocp | ||
} | ||
} |
10 changes: 10 additions & 0 deletions
10
data/data/bootstrap/openstack/files/usr/local/bin/fletcher8
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
#!/usr/libexec/platform-python | ||
import sys | ||
|
||
data = map(ord, sys.argv[1]) | ||
ckA = ckB = 0 | ||
|
||
for b in data: | ||
ckA = (ckA + b) & 0xf | ||
ckB = (ckB + ckA) & 0xf | ||
print((ckB << 4) | ckA ) |
24 changes: 24 additions & 0 deletions
24
data/data/bootstrap/openstack/files/usr/local/bin/get_vip_subnet_cidr
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
#!/usr/libexec/platform-python | ||
import sys | ||
import socket | ||
import struct | ||
|
||
vip = sys.argv[1] | ||
iface_cidrs = sys.argv[2].split() | ||
vip_int = struct.unpack("!I", socket.inet_aton(vip))[0] | ||
|
||
for iface_cidr in iface_cidrs: | ||
ip, prefix = iface_cidr.split('/') | ||
ip_int = struct.unpack("!I", socket.inet_aton(ip))[0] | ||
prefix_int = int(prefix) | ||
mask = int('1' * prefix_int + '0' * (32 - prefix_int), 2) | ||
subnet_ip_int_min = ip_int & mask | ||
subnet_ip = socket.inet_ntoa(struct.pack("!I", subnet_ip_int_min)) | ||
subnet_ip_int_max = subnet_ip_int_min | int('1' * (32 - prefix_int), 2) | ||
subnet_ip_max = socket.inet_ntoa(struct.pack("!I", subnet_ip_int_max)) | ||
sys.stderr.write('Is %s between %s and %s\n' % (vip, subnet_ip, subnet_ip_max)) | ||
if subnet_ip_int_min < vip_int < subnet_ip_int_max: | ||
subnet_ip = socket.inet_ntoa(struct.pack("!I", subnet_ip_int_min)) | ||
print('%s/%s' % (subnet_ip, prefix)) | ||
sys.exit(0) | ||
sys.exit(1) |
48 changes: 48 additions & 0 deletions
48
data/data/bootstrap/openstack/files/usr/local/bin/keepalived.sh.template
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,48 @@ | ||
#!/usr/bin/env bash | ||
set -e | ||
|
||
mkdir --parents /etc/keepalived | ||
|
||
# TODO(shadower): switch to the keepalived image from the release: | ||
# https://github.com/openshift/installer/pull/2025/files#diff-ce82c1d8a44f7dfc41dfc024085ccfeeR24 | ||
KEEPALIVED_IMAGE=quay.io/celebdor/keepalived:latest | ||
if ! podman inspect "$KEEPALIVED_IMAGE" &>/dev/null; then | ||
echo "Pulling release image..." | ||
podman pull "$KEEPALIVED_IMAGE" | ||
fi | ||
|
||
# TODO(shadower): at least some of these can be passed into this | ||
# template rather than discovered at runtime: | ||
API_DNS="$(sudo awk -F[/:] '/apiServerURL/ {print $5}' /opt/openshift/manifests/cluster-infrastructure-02-config.yml)" | ||
CLUSTER_NAME="$(awk -F. '{print $2}' <<< "$API_DNS")" | ||
API_VIP="{{ .InstallConfig.Platform.OpenStack.APIVIP }}" | ||
IFACE_CIDRS="$(ip addr show | grep -v "scope host" | grep -Po 'inet \K[\d.]+/[\d.]+' | xargs)" | ||
SUBNET_CIDR="$(/usr/local/bin/get_vip_subnet_cidr "$API_VIP" "$IFACE_CIDRS")" | ||
NET_MASK="$(echo "$SUBNET_CIDR" | cut -d "/" -f 2)" | ||
INTERFACE="$(ip -o addr show to "$SUBNET_CIDR" | head -n 1 | awk '{print $2}')" | ||
CLUSTER_DOMAIN="${API_DNS#*.}" | ||
|
||
# Virtual Router IDs. They must be different and 8 bit in length | ||
API_VRID=$(/usr/local/bin/fletcher8 "$CLUSTER_NAME-api") | ||
DNS_VRID=$(/usr/local/bin/fletcher8 "$CLUSTER_NAME-dns") | ||
|
||
export API_VIP | ||
export CLUSTER_NAME | ||
export INTERFACE | ||
export API_VRID | ||
export NET_MASK | ||
envsubst < /etc/keepalived/keepalived.conf.tmpl | sudo tee /etc/keepalived/keepalived.conf | ||
|
||
MATCHES="$(sudo podman ps -a --format "{{`{{.Names}}`}}" | awk '/keepalived$/ {print $0}')" | ||
if [[ -z "$MATCHES" ]]; then | ||
# TODO(bnemec): Figure out how to run with less perms | ||
podman create \ | ||
--name keepalived \ | ||
--volume /etc/keepalived:/etc/keepalived:z \ | ||
--network=host \ | ||
--privileged \ | ||
--cap-add=ALL \ | ||
"${KEEPALIVED_IMAGE}" \ | ||
/usr/sbin/keepalived -f /etc/keepalived/keepalived.conf \ | ||
--dont-fork -D -l -P | ||
fi |
18 changes: 18 additions & 0 deletions
18
data/data/bootstrap/openstack/systemd/units/keepalived.service
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
[Unit] | ||
Description=Manage node VIPs with keepalived | ||
Wants=network-online.target | ||
After=network-online.target | ||
|
||
[Service] | ||
WorkingDirectory=/etc/keepalived | ||
ExecStartPre=/usr/local/bin/keepalived.sh | ||
ExecStart=/usr/bin/podman start -a keepalived | ||
ExecStop=/usr/bin/podman stop -t 10 keepalived | ||
ConditionPathExists=!/etc/pivot/image-pullspec | ||
|
||
Restart=on-failure | ||
RestartSec=5 | ||
TimeoutStartSec=600 | ||
|
||
[Install] | ||
WantedBy=multi-user.target |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.