How to debug resetting into maintenance mode

[peter-facko]

Hello everyone

Me and my team are trying to install Talos on two small machines intended to serve as a staging cluster.

We were able to install Talos previously on version 1.11 but we didn't move to using it fully back then yet.

We are now trying to install 1.12. We hit quite a few errors with the new config kinds but right now I think we have figured that out.

The problem right now is that when we apply the config for the first time and the node resets, it starts up in maintenance mode. I think we boot from the installation disk however.

How can we debug this? We are just looking at quickly moving logs and fixed a few mistakes that way but we don't see anything helpful right now.

Do we capture the logs somehow? Or is resetting into maintenance mode some well known situation?

[rothgar]

You can capture remote logs with the logging config. See this video for an example https://youtube.com/shorts/z21SUfO5pjw?si=MjK-

Without having any logs or knowing what patches or system extensions you've applied it's not possible for us to give you adequate guidance to fix the problem.

It sounds like the machine is still booting from installation media instead of booting from the installed version of Talos.

[peter-facko]

Thank you for the reply @rothgar.

I have not collected the logs yet but it seems that the problem is at least partially with the new HostnameConfig (https://docs.siderolabs.com/talos/v1.12/reference/configuration/network/hostnameconfig).

We set a static hostname with the hostname: key. The generated config from talosctl gen config contains auto: stable and that seems to conflict. We use --config-patch to merge config in talosctl apply-config. I will try putting a patch that deletes the auto: key, but I find that behavior counterintuitive if that is the source of the problem.

[rothgar]

The old config is still supported but you can't mix config (old and new). Do you have examples of what patches you're trying to apply?

[peter-facko]

I have this

apiVersion: v1alpha1
kind: HostnameConfig
hostname: "a.cluster.example"

I see that the generated config in 1.12 contains

# ...
---
apiVersion: v1alpha1
kind: HostnameConfig
auto: stable # A method to automatically generate a hostname for the machine.

# # A static hostname to set for the machine.
# hostname: controlplane1
# hostname: controlplane1.example.org

[smira]

Add auto: off to your patch: https://docs.siderolabs.com/talos/v1.12/networking/configuration/hostname