Automatic upgrade and statefulset workloads

[smokfyz]

Description

Hello! There is limited information in the documentation regarding how node auto-upgrades function when using a database in Kubernetes or Longhorn. Could you please explain how auto-upgrades work in such a scenario? Just disable it or how?

[mysticaltech]

@smokfyz For k3s auto upgrades, nothing happens basically, the binary is replaced without even kube going down. Now for node upgrades, that's more invasive. Node upgrades provoke the nodes to be drained and uncordoned from the cluster for reboot, one after another, so super important to be in HA.

This I guess would be problematic unless you have longhorn correctly setup in an HA fashion, if you are in HA it should be seemless IMHO as longhorn would have distributed the data in duplicates across the cluster. Same for the statefulset it should not be a problem if configured in an HA fashion. But I must admit am not an expert in working with them.

@kube-hetzner/core Please correct me if I'm wrong 🙏

[mysticaltech]

Here's what I was able to get out of GPT-4:

Node upgrades and the management of StatefulSets in a High Availability (HA) Kubernetes cluster, particularly with automatic node upgrades, present a complex but manageable scenario. Here are some thoughts and considerations:

1. Importance of HA in Node Upgrades: In an HA cluster, automatic node upgrades are less likely to cause service disruptions. HA setups typically involve multiple replicas of nodes and pods, ensuring that even if one node is taken down for an upgrade, the others can continue to handle the workload. This redundancy is crucial for maintaining service availability.

2. Challenges with StatefulSets: StatefulSets, which are used for managing stateful applications (like databases), have their own complexities. They maintain a sticky identity for each of their pods. During node upgrades, care must be taken to ensure that the state (like persistent data) is not lost or corrupted. This requires a well-thought-out data replication and backup strategy.

3. Graceful Handling of Pod Eviction: During automatic node upgrades, pods are evicted as nodes are drained. For StatefulSets, Kubernetes tries to reschedule the pods to other nodes while maintaining their identity and connection to the right data volumes. Ensuring that other nodes have the capacity and configuration to accept these pods is essential.

4. Storage Considerations with Longhorn or Similar Solutions: If you're using a distributed storage solution like Longhorn, it's crucial to ensure that your storage layer is also highly available and resilient to node failures. Data should be replicated across multiple nodes so that if one node goes down during an upgrade, the data remains accessible.

5. Testing and Monitoring: Before implementing automatic upgrades in production, it’s vital to test the process in a staging environment. Monitoring tools should be in place to quickly identify and address any issues that arise during the upgrade process.

6. Version Compatibility and Rollback Plans: Ensure that the new versions of Kubernetes or other components are compatible with your existing workloads. Have a rollback plan in case the upgrade leads to unexpected issues.

7. Coordination and Scheduling: While automatic upgrades are convenient, it might be beneficial to have some control over the scheduling to ensure upgrades occur during low-traffic periods, thereby minimizing potential impacts.

In summary, while automatic node upgrades in an HA Kubernetes cluster offer many benefits in terms of reducing manual overhead and ensuring up-to-date systems, they require careful planning, robust infrastructure setup (especially for stateful applications), and thorough testing to ensure smooth operations without significant service disruptions.