[FEATURE] Optimize for Frequent Power-off/Power-On operating procedures

[rebeccazzzz]

Context
Running Harvester in edge or remote environments with either intermittent power or with devices needing to be turned off and moved to a new location frequently (sometimes daily).

Is your feature request related to a problem? Please describe.
Operators turning the cluster off and on are not highly technical with Kubernetes and thus can't be expected to troubleshoot stuck containers that don't come back online after startup.

update 20230824:
This issue is converted into an EPIC, it touchs k8s, csi, VM, Linux and more, one HEP is required.

A Harvester cluster is deployed in the rough sequence of:

OS -> rancherd -> rke2 -> k8s -> fleet -> harvester -> longhorn | monitoring | logging..., -> virtual-machine ...

To shutodown the cluster safely, follow the roughly reverse sequence seems reasonable.

A bunch of sub-stories are created to work on them.

Sub-stories in v1.3.0:

Bug fix and enhancement in v1.3.0:

A couple of bugs with various phenomena, but the root cause is same: after a node reboots, kubelet fails to UnmountVolume

• [BUG] VM got IO error after host restart #4633 (validate after rke2 is bumped to v1.27.10; kubelet fail to UnmountVolume)
• [BUG] After power off Harvester nodes then power on for days, all VMs display Stopping state and can't launch #4033
• [BUG] VM stuck in stopping state. Then after delete stuck in terminating phase #3539
• Is this a typical problem with this harvester that the rke2 clusters above the harvester no longer start (remains in pending state) after rebooting the harvester nodes following a power outage for example? version 1.2 #4789
• Add workaroud for 5109 docs#519 (The issue [BUG] Single Node Cluster Fails To Have VMs Come Back Post Restart w/ Storage Network Configured #5109 was observed when validating 4633, may have no chance to be fixed in Harvester v1.3.0, add it to Harvester document, after document PR is merged, this issue [BUG] Single Node Cluster Fails To Have VMs Come Back Post Restart w/ Storage Network Configured #5109 will be moved to v1.4.0)

Others:

• [BUG] Do not activate the LVM device on the harvester node #4674
• [BUG] Migrating (or simply stopping) VMs with LVM volumes causes Buffer I/O errors in kernel log of source host #3843 (possbible same root cause as 4674, will validate)
• [Doc] Add workaround for VM start button is not visible #4659

HEP PR:

• HEP: System robustness enhancement [CI SKIP] #4464

Continuous enhancment in v1.4.0+:

EPIC: #5007 [ENHANCEMENT] Continuous enhancement on system robustness and resilience

• Enhance VM backup and restore #4447
• Enhance k8s (etcd) backup and restore #4450
• Enhance 3-rd party CSI stopping #4449
• Enhance NODE level stop/maintain and shutdown #4451
• Enhance UI with cluster shutdown menu and status display #4452
• Enhance Longhorn stopping and recovering #4448

Those ISSUEs/PRs could be checked & algined as well:
#3902
#3263
harvester/node-manager#15

[harvesterhci-io-github-bot]

Pre Ready-For-Testing Checklist

• If labeled: require/HEP Has the Harvester Enhancement Proposal PR submitted?
  The HEP PR is at:

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is at:

• Have the backend code been merged (harvester, harvester-installer, etc) (including backport-needed/*)?
  The PR is at:

  • Does the PR include the explanation for the fix or the feature?

  • Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
    The PR for the YAML change is at:
    The PR for the chart change is at:

• If labeled: area/ui Has the UI issue filed or ready to be merged?
  The UI issue/PR is at:

• If labeled: require/doc, require/knowledge-base Has the necessary document PR submitted or merged?
  The documentation/KB PR is at:

• If NOT labeled: not-require/test-plan Has the e2e test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue?

  • The automation skeleton PR is at:
  • The automation test case PR is at:

• If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
  The compatibility issue is filed at:

[harvesterhci-io-github-bot]

Automation e2e test issue: harvester/tests#664

[w13915984028]

New updated scenario:

Run 5+ VMs on the cluster (1~3 Nodes), stop all nodes at same time, then restart the cluster at certain time. Some VMs may stuck / take quite a long time to restart:

• Some may be related issues:
  [BUG] After power off Harvester nodes then power on for days, all VMs display Stopping state and can't launch #4033
  [BUG] VM stuck in stopping state. Then after delete stuck in terminating phase #3539
  [FEATURE] rebuild last healthy replica during node drain #3378
  [BUG] VM got IO error after host restart #4633
  [BUG] vm stuck in loop toggle between "pause"/"unpause" #4828
  [BUG] VM is stuck in terminating after a reboot  #994 (comment)

[w13915984028]

The planned optimization and bug fix in v1.3.0 have been done, and the continuous enhancement is tracked new epic #5007 in Harvester v1.4.0. Close this issue now.