Skip to content

Commit

Permalink
WIP openstack: remove the service VM
Browse files Browse the repository at this point in the history 
This is part of the work to remove the service VM from the openstack
architecture. This relies on the coredns/mdns and haproxy static pods
setup in: openshift/machine-config-operator/pull/740

TODO(shadower): clean up the commit message, explain what we're doing,
why we're doing it, all the external dependencies (MCO, CAPO) the
current status and what still needs to be done.

Below are the messages of all the commits squashed to get this in:

openstack: remove service vm

Run haproxy on 7443 with NAT rule forward to 6443

openstack: add clustervars that can be sourced by static pods

hacks: integrate changes from pull/1808

hacks: start haproxy with only bootstrap node in backends

don't use /tmp for clustervars

hacks: don't redirect traffic from the cluster CIDR

hacks: open master SGs to internet

Set domain search to cluster domain

So that nodes can resolve other nodes in the cluster using short names.

Remove unused master_port_names tfvar

WIP Some progress on getting Ignition via IP

The LbFloatingIP is now used as the predictible address for API. It
points to the bootstrap node first, then is moved to the first master
upon bootstrap removal. As a consequence, the LbFloatingIP becomes
a mandatory parameter for the installer on OpenStack platform.

This patch also cleans up the network architecture a little bit by
removing the subnet complexity for the deployed nodes. There is now
only one subnet for all the provisioned nodes.

Some cleanup

Stop setting selinux permissive on master nodes

Use bootstrap node's default hosts file

Unbreak other platforms for ignition retrieval

Make the ignition retrieval via IP address specific to the OpenStack
platform.

bootstrap: add switch-api-endpoint service

There is a potential cycle where the temporary bootstrap control plane
gets torn down and the API endpoint on the bootstrap node points to
itself (via the floating IP) rather than the masters.

The installer is waiting for the bootstrapping to be completed, but the
`progress` service is unable to send the bootstrap-complete event,
because the API server is no longer running on the bootstrap node (which
still has the FIP attached).

Therefore, the installer never runs the bootstrap destroy terraform
actions and the FIP is stuck on the bootstrap node.

This adds a new service that waits until the `bootkube` and `openshift`
services finish (just like `progress.service` does), but then creates an
`/etc/hosts` entry for the API endpoints so that `progress` can
communicate with the master control plane.

Run haproxy from MCO

Move the bootstrap API endpoint after etcd

That should fix the connection issues I've been seeing. But it's still
not enough for some reason.
  • Loading branch information
@tomassedovic
trown authored and tomassedovic committed Jul 9, 2019
1 parent 1cb12b5 commit 73fc1ba
Show file tree
Hide file tree
Showing 18 changed files with 215 additions and 583 deletions.
1 change: 1 addition & 0 deletions data/data/bootstrap/files/usr/local/bin/bootkube.sh.template
View file
Expand Up @@ -275,6 +275,7 @@ echo "etcd cluster up. Killing etcd certificate signer..."

podman rm --force etcd-signer
rm --force /etc/kubernetes/manifests/machineconfigoperator-bootstrap-pod.yaml
touch /opt/openshift/etcd-bootstrap.done

echo "Starting cluster-bootstrap..."

Expand Down
98 changes: 81 additions & 17 deletions data/data/openstack/bootstrap/main.tf
View file
Expand Up @@ -18,34 +18,47 @@ data "ignition_config" "redirect" {

files = [
data.ignition_file.hostname.id,
data.ignition_file.bootstrap_ifcfg.id,
data.ignition_file.dns_conf.id,
data.ignition_file.dhcp_conf.id,
data.ignition_file.switch_api_endpoint.id,
]

systemd = [
data.ignition_systemd_unit.switch_api_endpoint.id,
]
}

data "ignition_file" "bootstrap_ifcfg" {
data "ignition_file" "dhcp_conf" {
filesystem = "root"
mode = "420" // 0644
path = "/etc/sysconfig/network-scripts/ifcfg-eth0"
mode = "420"
path = "/etc/NetworkManager/conf.d/dhcp-client.conf"

content {
content = <<EOF
DEVICE="eth0"
BOOTPROTO="dhcp"
ONBOOT="yes"
TYPE="Ethernet"
PERSISTENT_DHCLIENT="yes"
DNS1="${var.service_vm_fixed_ip}"
PEERDNS="no"
NM_CONTROLLED="yes"
[main]
dhcp=dhclient
EOF
}
}

data "ignition_file" "dns_conf" {
filesystem = "root"
mode = "420"
path = "/etc/dhcp/dhclient.conf"

content {
content = <<EOF
send dhcp-client-identifier = hardware;
prepend domain-name-servers ${var.master_vm_fixed_ip};
EOF

}
}

data "ignition_file" "hostname" {
filesystem = "root"
mode = "420" // 0644
path = "/etc/hostname"
mode = "420" // 0644
path = "/etc/hostname"

content {
content = <<EOF
Expand All @@ -55,8 +68,59 @@ EOF
}
}

data "ignition_file" "switch_api_endpoint" {
filesystem = "root"
mode = "493" // 0755
path = "/usr/local/bin/switch-api-endpoint.sh"

content {
content = <<EOF
#!/usr/bin/env bash
set -eu
wait_for_existence() {
while [ ! -e "$${1}" ]
do
sleep 5
done
}
echo "Waiting for etcd bootstrap to complete..."
wait_for_existence /opt/openshift/etcd-bootstrap.done
echo "Switching bootstrap's API address to a master node"
echo "${var.master_vm_fixed_ip} api-int.${var.cluster_domain} api.${var.cluster_domain}" >> /etc/hosts
EOF

}
}

data "ignition_systemd_unit" "switch_api_endpoint" {
name = "switch-api-endpoint.service"
enabled = true

content = <<EOF
[Unit]
Description=Switch the bootstrap API to a master node. This will enable `progress.service` to send the boostrap-complete event.
# Workaround for https://github.com/systemd/systemd/issues/1312
Wants=bootkube.service openshift.service
After=bootkube.service openshift.service
[Service]
ExecStart=/usr/local/bin/switch-api-endpoint.sh
Restart=on-failure
RestartSec=5s
[Install]
WantedBy=multi-user.target
EOF
}


data "openstack_images_image_v2" "bootstrap_image" {
name = var.image_name
name = var.image_name
most_recent = true
}

Expand All @@ -65,9 +129,9 @@ data "openstack_compute_flavor_v2" "bootstrap_flavor" {
}

resource "openstack_compute_instance_v2" "bootstrap" {
name = "${var.cluster_id}-bootstrap"
name = "${var.cluster_id}-bootstrap"
flavor_id = data.openstack_compute_flavor_v2.bootstrap_flavor.id
image_id = data.openstack_images_image_v2.bootstrap_image.id
image_id = data.openstack_images_image_v2.bootstrap_image.id

user_data = data.ignition_config.redirect.rendered

Expand Down
10 changes: 9 additions & 1 deletion data/data/openstack/bootstrap/variables.tf
View file
Expand Up @@ -33,7 +33,15 @@ variable "bootstrap_port_id" {
description = "The subnet ID for the bootstrap node."
}

variable "service_vm_fixed_ip" {
variable "bootstrap_ip" {
type = string
}

variable "lb_floating_ip" {
type = string
}

variable "master_vm_fixed_ip" {
type = "string"
description = "Fixed IP for a master node to provide DNS to bootstrap during clustering."
}
66 changes: 26 additions & 40 deletions data/data/openstack/main.tf
View file
Expand Up @@ -22,69 +22,53 @@ provider "openstack" {
user_name = var.openstack_credentials_user_name
}

module "service" {
source = "./service"

swift_container = openstack_objectstorage_container_v1.container.name
cluster_id = var.cluster_id
cluster_domain = var.cluster_domain
image_name = var.openstack_base_image
flavor_name = var.openstack_master_flavor_name
ignition = var.ignition_bootstrap
lb_floating_ip = var.openstack_lb_floating_ip
service_port_id = module.topology.service_port_id
service_port_ip = module.topology.service_port_ip
master_ips = module.topology.master_ips
master_port_names = module.topology.master_port_names
bootstrap_ip = module.topology.bootstrap_port_ip
}

module "bootstrap" {
source = "./bootstrap"

swift_container = openstack_objectstorage_container_v1.container.name
cluster_id = var.cluster_id
cluster_domain = var.cluster_domain
image_name = var.openstack_base_image
flavor_name = var.openstack_master_flavor_name
ignition = var.ignition_bootstrap
bootstrap_port_id = module.topology.bootstrap_port_id
service_vm_fixed_ip = module.topology.service_vm_fixed_ip
swift_container = openstack_objectstorage_container_v1.container.name
cluster_id = var.cluster_id
cluster_domain = var.cluster_domain
image_name = var.openstack_base_image
flavor_name = var.openstack_master_flavor_name
ignition = var.ignition_bootstrap
bootstrap_port_id = module.topology.bootstrap_port_id
lb_floating_ip = var.openstack_lb_floating_ip
master_vm_fixed_ip = module.topology.master_ips[0]
bootstrap_ip = module.topology.bootstrap_port_ip
}

module "masters" {
source = "./masters"

base_image = var.openstack_base_image
bootstrap_ip = module.topology.bootstrap_port_ip
cluster_id = var.cluster_id
cluster_domain = var.cluster_domain
flavor_name = var.openstack_master_flavor_name
instance_count = var.master_count
lb_floating_ip = var.openstack_lb_floating_ip
master_ips = module.topology.master_ips
master_port_ids = module.topology.master_port_ids
master_port_names = module.topology.master_port_names
user_data_ign = var.ignition_master
service_vm_fixed_ip = module.topology.service_vm_fixed_ip
base_image = var.openstack_base_image
bootstrap_ip = module.topology.bootstrap_port_ip
cluster_id = var.cluster_id
cluster_domain = var.cluster_domain
flavor_name = var.openstack_master_flavor_name
instance_count = var.master_count
lb_floating_ip = var.openstack_lb_floating_ip
master_ips = module.topology.master_ips
master_port_ids = module.topology.master_port_ids
master_port_names = module.topology.master_port_names
user_data_ign = var.ignition_master
master_sg_ids = concat(
var.openstack_master_extra_sg_ids,
[module.topology.master_sg_id],
)
}

# TODO(shadower) add a dns module here

module "topology" {
source = "./topology"

cidr_block = var.machine_cidr
cluster_id = var.cluster_id
cluster_domain = var.cluster_domain
external_network = var.openstack_external_network
external_network_id = var.openstack_external_network_id
masters_count = var.master_count
lb_floating_ip = var.openstack_lb_floating_ip
trunk_support = var.openstack_trunk_support
bootstrap_dns = var.bootstrap_dns
}

resource "openstack_objectstorage_container_v1" "container" {
Expand All @@ -93,9 +77,11 @@ resource "openstack_objectstorage_container_v1" "container" {
# "kubernetes.io/cluster/${var.cluster_id}" = "owned"
metadata = merge(
{
"Name" = "${var.cluster_id}-ignition-master"
"Name" = "${var.cluster_id}-ignition"
"openshiftClusterID" = var.cluster_id
},
# FIXME(mandre) the openstack_extra_tags should be applied to all resources
# created
var.openstack_extra_tags,
)
}
Expand Down
3 changes: 2 additions & 1 deletion data/data/openstack/masters/main.tf
View file
Expand Up @@ -17,7 +17,6 @@ data "ignition_file" "hostname" {
content = <<EOF
master-${count.index}
EOF

}
}

Expand Down Expand Up @@ -46,6 +45,7 @@ data "ignition_config" "master_ignition_config" {
element(data.ignition_file.hostname.*.id, count.index),
data.ignition_file.clustervars.id,
]

}

resource "openstack_compute_instance_v2" "master_conf" {
Expand All @@ -65,6 +65,7 @@ resource "openstack_compute_instance_v2" "master_conf" {
}

metadata = {
# FIXME(mandre) shouldn't it be "${var.cluster_id}-master-${count.index}" ?
Name = "${var.cluster_id}-master"
# "kubernetes.io/cluster/${var.cluster_id}" = "owned"
openshiftClusterID = var.cluster_id
Expand Down
4 changes: 0 additions & 4 deletions data/data/openstack/masters/variables.tf
View file
Expand Up @@ -50,7 +50,3 @@ variable "master_port_names" {
variable "user_data_ign" {
type = string
}

variable "service_vm_fixed_ip" {
type = string
}

0 comments on commit 73fc1ba

Please sign in to comment.