[BACKPORT][v1.5.4][BUG] Deadlock is possible in v1.6.0 instance manager

[github-actions]

backport #7919

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at: [BUG] Deadlock is possible in v1.6.0 instance manager #7919 (comment). However, they probably require a modified version of instance-manager to be reliable. I tested it like Don't send on unbuffered channel while holding a lock longhorn-instance-manager#397 (comment), but for QA purposes, I suggest we just ensure end-to-end/integration tests pass.

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is at: [BUG] Volume attach/detach/delete operations stuck in version 1.6.0 #7915 (comment).

• Does the PR include the explanation for the fix or the feature?

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at: Don't send on unbuffered channel while holding a lock (backport #397) longhorn-instance-manager#400.

[roger-ryao]

Verified on v1.5.x-head 20240219

• longhorn v1.5.x-head 3aa3ce9
• longhorn v1.5.x-head longhorn/longhorn-instance-manager@8865978

The test steps

1. [BUG] Deadlock is possible in v1.6.0 instance manager #7919 (comment)
2. Examine the regression test results for v1.5.x-head to identify any outstanding issues.

Result Passed