[IMPROVEMENT] Recreate instance manager pod for v2 volume when spdk_tgt is dead

[derekbit]

Is your improvement request related to a feature? Please describe (👍 if you like this request)

spdk_tgt might be somehow dead due to

[ 6720.296447] reactor_0[11215]: segfault at 10 ip 00000000004fb2da sp 00007ffe5c445cb0 error 4 in spdk_tgt[400000+3d5000]
[ 6720.296466] Code: 48 8b b7 f8 02 00 00 48 8b 40 18 48 8b 18 48 85 f6 74 17 48 8b 97 00 03 00 00 e8 51 ff ff ff 48 c7 85 f8 02 00 00 00 00 00 00 <8b> 43 10 3b 43 14 0f 83 8a 00 00 00 83 c0 01 89 43 10 48 8b 03 48

Instance manager pod should detect it and terminate itself.

Describe the solution you'd like

Describe alternatives you've considered

Additional context

[innobead]

cc @DamiaSan

[derekbit]

segfault in spdk_tgt needs further investigation and @DamiaSan's help.

The PRs I submitted are improving the resilience.

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

• Fresh installation
  1. Enable v1-data-engine and v2-data engine
  2. Check instance-manager pods for v1 and v2 data engines works as expected
  3. Go to one of instance-manager pods for v2 data engine. Then, killall -9 spdk_tgt
  4. Wait for a while, the instance-manager pod should be recreated. Other pods should not be impacted.
• Upgrade

  1. Install Longhorn v1.5.3

  2. Create some v1 volumes

  3. Upgrade Longhorn to master-head

  4. Enable v1-data-engine and v2-data engine

  5. Old and new instance-manager pods works as expected

  6. Go to one of instance-manager pods for v2 data engine. Then, killall -9 spdk_tgt

  7. Wait for a while, the instance-manager pod should be recreated. Other pods should not be impacted.

  8. Detach v1 volumes

  9. Old instance-manager pods should be deleted

  10. Attach v1 volumes and should work

• Does the PR include the explanation for the fix or the feature?

Update liveness probe of instance-manager pods.

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at

longhorn/longhorn-manager#2428
longhorn/longhorn-spdk-engine#87
longhorn/longhorn-instance-manager#356

• Which areas/issues this PR might have potential impacts on?
  Area: instance manager pod liveness probe. instance manager for v2 data engine.
  Issues

[chriscchien]

Verified pass on longhorn master (longhorn-manager 325252) test steps

After kill spdk_tgt in v2 volume instance-manager pod, the instance-manager pod will recreate and after pod ready, all volumes worked well. (tested on freash installed v1.6.0-dev and upgrade from v1.5.3)