Allow overriding/specifying IRONIC_IP/IRONIC_URL_HOST

[lentzi90]

It is currently possible to set either the PROVISIONING_INTERFACE or PROVISIONING_IP. If PROVISIONING_IP is set, this IP is assumed to be associated with some interface and Ironic will wait until it can see it there. If PROVISIONING_INTERFACE is set, we check what IP is associated with it and assume that this is the IRONIC_IP and IRONIC_URL_HOST.

Ref:

ironic-image/scripts/ironic-common.sh

Lines 43 to 72 in d62a9b7

	wait_for_interface_or_ip()
	{
	# If $PROVISIONING_IP is specified, then we wait for that to become available on an interface, otherwise we look at $PROVISIONING_INTERFACE for an IP
	if [[ -n "$PROVISIONING_IP" ]]; then
	# Convert the address using ipcalc which strips out the subnet. For IPv6 addresses, this will give the short-form address
	IRONIC_IP="$(ipcalc "${PROVISIONING_IP}" | grep "^Address:" | awk '{print $2}')"
	export IRONIC_IP
	until grep -F " ${IRONIC_IP}/" <(ip -br addr show); do
	echo "Waiting for ${IRONIC_IP} to be configured on an interface"
	sleep 1
	done
	else
	until [[ -n "$IRONIC_IP" ]]; do
	echo "Waiting for ${PROVISIONING_INTERFACE} interface to be configured"
	IRONIC_IP="$(ip -br add show scope global up dev "${PROVISIONING_INTERFACE}" | awk '{print $3}' | sed -e 's%/.*%%' | head -n 1)"
	export IRONIC_IP
	sleep 1
	done
	fi
	# If the IP contains a colon, then it's an IPv6 address, and the HTTP
	# host needs surrounding with brackets
	if [[ "$IRONIC_IP" =~ .*:.* ]]; then
	export IPV=6
	export IRONIC_URL_HOST="[$IRONIC_IP]"
	else
	export IPV=4
	export IRONIC_URL_HOST="$IRONIC_IP"
	fi
	}

I would like to expose Ironic through a Service of type LoadBalancer instead of using host network. When doing this, the load balancer IP will not be directly associated with any interface in the container. This means that I cannot set the PROVISIONING_IP since Ironic would then wait indefinitely to see this IP on some interface (which will never happen). Instead I set the PROVISIONING_INTERFACE. This works great and I can reach both Ironic and Inspector when curling the load balancer IP (e.g. 192.168.222.200).

However, due to the snippet above, Ironic and Inspector will be configured to try to reach each other using the cluster network IP of the Pod (e.g. 10.244.0.13). This IP is volatile and not something that would be in the certificate when using TLS, so communication breaks down.

What can we do about this? Is it something that would go away together with the Inspector anyway? Maybe not worth doing anything at this point then. Or should we make it possible to override the IRONIC_IP and/or IRONIC_URL_HOST?

If this sounds interesting and you would like to play with it, try this:

1. Clone https://github.com/lentzi90/playground/tree/ironic-loadbalancer#metal3 (use the branch ironic-loadbalancer)
2. Run ./Metal3/dev-setup.sh
3. Wait for all pods to be up
4. Curl the APIs:
   1. curl https://192.168.222.200:5050 -k
   2. curl https://192.168.222.200:6385 -k
5. Try creating a BMH to see the inspection error: NUM_BMH=1 ./Metal3/create-bmhs.sh

[dtantsur]

Is it something that would go away together with the Inspector anyway?

Yes, but we may hit similar issues with JSON RPC. Which is, whoever, its own ball of wax.

[lentzi90]

Is it something that would go away together with the Inspector anyway?

Yes, but we may hit similar issues with JSON RPC. Which is, whoever, its own ball of wax.

Could you elaborate? I'm afraid I'm lost 😅

[dtantsur]

For JSON RPC to work, Ironic instances have to be able to talk to each other. That requires every Ironic to know its own IP or host name.

[Rozzii]

/triage accepted
/help

[metal3-io-bot]

@Rozzii:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/triage accepted
/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[lentzi90]

Based on the discussion in the last community meeting, I'm adding some notes here so I don't forget.

• Ironic can do internal load balancing when running multiple instances
• We could have an external load balancer to provide a fixed endpoint for clients (this was what I was trying to do)
• Perhaps we can get some ideas from how etcd is deployed. It seems to be similar in that it also has one port for clients and another for peers. It needs to know about all peers, while clients only need to know about the single endpoint for clients.

[metal3-io-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues will close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[lentzi90]

I think we can close this as fixed in #476