Add bootstate endpoint to registry server

[Nuckal777]

Proposed Changes

• This is a straightforward implementation, which adds a Kubernetes Client to the HTTP server. If this is not desired, we should align on how to transport the event of calling the endpoint into the controller.
• In contrast to Define /bootstate call back endpoint in manager to track if Server successfully booted #399 the systemUUID is moved into the JSON payload, similar to the /register endpoint.

Fixes #399.

[afritzler]

Adding a k8sClient to the registry server will make it harder in the future to factor out the registry and deploy it somewhere were it might not have access to the metal-operator api server. How about instead the ServerReconciler probes the /bootstate endpoint periodically since we are doing a periodic retry on all Server objects?

[Nuckal777]

How about instead the ServerReconciler probes the /bootstate endpoint periodically since we are doing a periodic retry on all Server objects?

This would introduce a gap in which the bootstate can be lost:

• Server boots up, calls the bootstate endpoint and receives an HTTP 200.
• Some time passes, while the bootstate only resides in memory
• Server controller reconciles the Server object and applies the condition

Restarting the registry in the second step would loose the bootstate event.

[hardikdr]

Thanks for the PR @Nuckal777, dropped some nit inline.

Thinking about it a little more, I’d propose a following holistic way of handling the first-boot flow. Instead of tying it to a single signal, we could make it more flexible, for example, let the boot-operator set conditions like IPXEScriptFetched and IgnitionDataFetched, and have the /bootstate POST update a BootStateReceived condition on the ServerBootConfig.

Then, the metal-operator could decide which of these conditions to treat as the actual boot completion using a boot-completion-condition flag. This would also make it easier to support things like NetBootOnce and NetBootAlways policies on the ServerClaim side later even when /bootstate POST call is not configured in the Ignition. Wdyt?

[hardikdr]

It would be interesting to explore the trade offs between having this condition on Server vs ServerBootConfig CR. I am somehow leaning more towards ServerBootConfig.

[Nuckal777]

Hmm, ServerBootConfiguration might be more fitting as their lifetimes are bound to a ServerClaim and the discovery boot. 🤔

[hardikdr]

Perfect, I would say let's already move it to ServerBootConfig, and we can also discuss this in our PR review roulette soon.

[Nuckal777]

Instead of tying it to a single signal, we could make it more flexible...

Agree, iirc it was part of the initial discussion on the topic that the owner of a ServerClaim could choose what is considered to be a successful network boot.

[hardikdr]

Instead of tying it to a single signal, we could make it more flexible...

Agree, iirc it was part of the initial discussion on the topic that the owner of a ServerClaim could choose what is considered to be a successful network boot.

Sure, having a knob on the ServerClaim would allow configuring it per Claims then, I would also consider tradeoffs against having a common single flag in the metal-operator.

[hardikdr]

Tests seem to be failing.
Looks good otherwise.

[defo89]

I think it looks good, also find the suggestions from @hardikdr valuable.

For the future we could consider adding BootStateReceivedTimeout for cases registry fails to receive the boot state from server (issues with registry or server failed to boot), to retry redeploying a server.