KVM: add storage-heartbeat fencing fallback when OOBM fence fails (host stuck in Fencing on total power loss)

[nikolauseppinger]

Description

When a KVM host suffers a complete power failure, its BMC (IPMI/iDRAC/iLO)
usually loses power together with the host. The Host HA framework's fence
operation (KVMHAProvider.fence()) relies solely on out-of-band management
(PowerOperation.OFF), so fencing can never succeed in this scenario:

• The FSM has no exit from Fencing except Fenced or Disabled, so the
  host retries fencing forever (RetryFencing).
• FenceTask.processResult() only calls
  oldHighAvailabilityManager.scheduleRestartForVmsOnHost() on a successful
  fence, so HA-enabled VMs are never restarted.
• HAManagerImpl.isVMAliveOnHost() returns true for every state except
  Fenced, which also blocks the legacy VM HA investigation path.

The result: on a total power failure — arguably the most common
total-failure scenario HA exists for — all HA-enabled VMs stay down
indefinitely until an operator intervenes (e.g. via declareHostAsDegraded).

Related reports: #12921, #12185.

At the point fence() is invoked, CloudStack has already gathered strong
evidence that the host is dead:

1. Health checks failed (agent gone),
2. The configured number of activity checks found no VM disk activity on any
   pool (CheckVMActivityOnStoragePoolCommand),
3. Neighbouring hosts positively report the host's storage heartbeat as
   expired — the same evidence the legacy KVMFencer has always accepted as
   sufficient proof of fencing.

Change

This PR adds an opt-in fallback, disabled by default:

• New cluster-scope setting kvm.ha.fence.on.storage.heartbeat
  (default false).
• When the OOBM fence operation fails (unsuccessful response, OOBM disabled,
  or an exception such as an unreachable BMC), the host is considered fenced
  only if the neighbouring hosts positively report its storage heartbeat
  as expired (Status.Down from KVMHostActivityChecker.getHostAgentStatus(),
  which requires neighbour confirmation via CheckOnHostCommand).
• The host is never considered fenced when any check still sees it alive,
  when no neighbour can confirm (Disconnected/Unknown), or when the
  heartbeat check itself fails — absence of evidence never counts as evidence
  of death.
• With the setting disabled (default), behaviour is completely unchanged.

The heartbeat semantics follow the existing checks, including
kvm.ha.fence.host.if.heartbeat.fails.on.storage.

Safety considerations

Fencing exists to guarantee the host can no longer write to shared storage
before VMs are restarted elsewhere. The fallback only fires after three
independent confirmations (failed health checks, no VM disk activity over the
whole activity-check window, expired storage heartbeat confirmed by
neighbours), which is strictly more evidence than the legacy KVMFencer
requires for the same decision. A transient BMC outage on a live host does
not trigger it: a live host keeps writing its heartbeat, so neighbours will
not report Down.

Types of changes

• Enhancement (improves an existing feature and functionality)
• Bug fix (non-breaking change which fixes an issue)

Bug Severity

• Major

How Has This Been Tested?

• 6 new unit tests in KVMHostHATest covering all fence paths (OOBM success,
  OOBM failure with the setting disabled/enabled, heartbeat dead/alive, OOBM
  exception with and without fallback); 10/10 tests pass.
• mvn -pl plugins/hypervisors/kvm -am test builds green including checkstyle.
• Scenario observed on 4.22.1.0 (KVM on RHEL 9.8, SharedMountPoint primary
  storage, iDRAC OOBM): host power loss took down the iDRAC, host remained in
  Fencing with all HA VMs down; neighbour heartbeat checks had reported the
  host dead 62 seconds after the failure.