[BUG] Node disconnection test failed

[yangchiu]

Describe the bug (🐛 if you encounter this issue)

In Node disconnection test case 1, the desired behavior is:

If there is data writing during the disconnection, 
due to the engine process not able to talk with other replicas, 
the engine process will mark all other replicas as ERROR.

The volume will remain detached, and all replicas remain in error state after node network connection back.

But in v1.4.1-rc1, network disconnection will not cause the replica on the attached node becoming error state, instead it remains healthy, and after node network connection back, replicas on other nodes can be rebuilt from this healthy replica, and finally all the replicas are in healthy state, which is not the expected behavior.

Need to confirm whether the behavior changing is expected.

To Reproduce

manually execute Node disconnection test case 1

Expected behavior

A clear and concise description of what you expected to happen.

Log or Support bundle

If applicable, add the Longhorn managers' log or support bundle when the issue happens.
You can generate a Support Bundle using the link at the footer of the Longhorn UI.

Environment

• Longhorn version:
• Installation method (e.g. Rancher Catalog App/Helm/Kubectl):
• Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version:
  • Number of management node in the cluster:
  • Number of worker node in the cluster:
• Node config
  • OS type and version:
  • CPU per node:
  • Memory per node:
  • Disk type(e.g. SSD/NVMe):
  • Network bandwidth between the nodes:
• Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal):
• Number of Longhorn volumes in the cluster:

Additional context

Add any other context about the problem here.

[innobead]

@yangchiu how long the disconnection was?

Case one is for 100 seconds for the disconnection, then what we would expect should be replicas marked as error already

cc @derekbit @shuo-wu @c3y1huang

[yangchiu]

@yangchiu how long the disconnection was?

Case one is for 100 seconds for the disconnection, then what we would expect should be replicas marked as error already

cc @derekbit @shuo-wu @c3y1huang

100 seconds.

Following the same test steps, v1.4.0 doesn't have this problem. The behavior is same as the description in the test case.

[derekbit]

The behavior change is due to the commit longhorn/longhorn-manager#1691.

In master-head/v1.4.1-rc1, disable auto-salvage in the following scenarios.

Scenario 1: one of the replicas is on the attached node.

• Attached node is disconnected
• The replica is on the attached node becomes unknown. The connection between the engine and the replica still works, so the unknown state should be expected.
• The replica becomes running after the network connection is back. Then, other replicas are rebuilt from the replica on the attached node.
• In the end, all replicas are running.

Scenario 2: none of the replicas is on the attached node.

• Attached node is disconnected
• All replicas become stopped after the network connection is back.

@yangchiu Can you help check the behaviors again? Thank you.

[innobead]

If confirming the above test cases, then we need to update the test doc. cc @longhorn/qa

[yangchiu]

Yes, current behavior follows #5476 (comment), and the test case needs to be updated.

[innobead]

@yangchiu It seems behavior change, so just close this if all is good, and also update the test cases accordingly. Thank.

cc @longhorn/qa

[yangchiu]

Manual test updated in longhorn/longhorn-tests#1278, we can close this one.