IPA on baremetal server is unable to reach ironic's endpoint - ENETUNREACH

[agewagra]

Steps to reproduce the bug:
follow the deployment steps for ironic and baremetal operator. Use the metal3 ironic-image for deploying ironic. https://book.metal3.io/quick-start.
After deployment, create a baremetalhost for a baremetal server using the yaml file.

Environment:
Deployed on an openshift cluster v4.13 - dual stack environment.

What did you expect to happen:
The IPA on RAMdisk of baremetal server should reach ironic's endpoint in prebooting of the inspection phase and BMH status should be available.

Anything else you would like to add:
We are using a dual stack environment on openshift to make deployments. The baremetal server that we want to provision needs the virtual boot image from ironic to be hosted on ipv6 address. So we gave an ipv6 address for ironic_external_ip. Rest all are ipv4 addresses.

• Baremetal Operator version: latest
• Environment (metal3-dev-env or other): Openshift kubernetes cluster - 4.13

Bug:
IPA is trying to perform lookup for node on ironic during inspection. But it is unable to reach the end point.

[metal3-io-bot]

This issue is currently awaiting triage.
If Metal3.io contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.
The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[matthewei]

Maybe you can set ironic-endpoint "https://ip:port" to have a try.

[agewagra]

Hi,
this is the ironic endpoint that we are using: http://10.59.52.241:6385, which I think is the same as what you are saying.
Correct me if I am wrong

[dtantsur]

Hi! Just to clarify: you're using OpenShift as Kubernetes, but you're not using a built-in Metal3 (that is deployed automatically if you have platform=baremetal)? You're using upstream BMO, Ironic and IPA of the latest versions? (I wonder why, but okay). Generally, we don't debug OpenShift-specific issues in the context of the upstream Metal3 community, but if so:

This is a networking problem, we cannot debug it just from the description. You may need to log into the ramdisk (figuring out what it actual IP is) and check the networking. If you're using a provisioning network and it's IPv6-only, that's the expected outcome, for example.

[agewagra]

Hi, thanks for the response.
Yeah we are using Openshift as kuberenetes and yes we are using the latest versions of BMO, Ironic and IPA.

In the iso image on ramdisk, we confirmed that network data is added in openstack/latest/network_data.json file. Our environment does not support DHCP. We used preprovisioningNetworkData in BMH to add network data. This is supposed to work in DHCP less environment according to https://github.com/metal3-io/metal3-docs/blob/main/design/baremetal-operator/image-builder-integration.md. But the IPA is still waiting for DHCP to assign IP, despite giving the static network data. And since there is no ip that is assigned to the BareMetalServer, it is unable to reach back ironic's endpoint.

Is there any way to disable waiting for DHCP from IPA or generate custom IPA without DHCP waiting?

[dtantsur]

Is it a duplicate of metal3-io/metal3-docs#416 then? Could you close one of them?

[Rozzii]

@agewagra Have you built your own custon IPA or are you using the default from upstream?
The default upstream IPA doesn't contain the tooling required for network configuration so you would need most likely "Glean" or Cloud-init.
/triage needs-information

[agewagra]

Yeah we were using default, but realised we should build the custom IPA.
Can we build a custom IPA with RHCOS as base image?

[Rozzii]

Yeah we were using default, but realised we should build the custom IPA. Can we build a custom IPA with RHCOS as base image?

Technically by creating custom "elements" for the IPA builder or for OpenStack disk image builder you can build IPA out of any base image. IPA builder is based on Openstack Disk Image Builder (DIB) and DIB is a modular tool so you can build your own custom workflows. I would expect that DIB has very good RH distro support by default but I have not used RHCOS. Maybe @dtantsur or @elfosardo is more up to date .

[agewagra]

Yes, thank you. Added Simple-init element to get a dhcp less ramdisk with centOS as base image. But even then on the baremetal server, the networkData is not being applied.