[BUG] Continuously rebuild when auto-balance==least-effort and existing node becomes unschedulable

[docbobo]

Describe the bug

I was just noticing some weird interactions beet auto-balance "least-effort" and unschedulable replicas when I had to cordon a few of my nodes. Here's a quick description:

I have a volume with 3 replicas, with auto-balance configure to "ignored", so that it would fall back to my system default of least effort. Each of the replicas is assigned to a different zone. When I had to cordon a few of the nodes, all of the nodes from one of the zone became unschedulable and there were literally only two zones left. However, the replica on the unschedulable node was still running. So far, so good.

Longhorn then started to build a fourth replica in one of the two regions already being used. When it was done, it deleted one of the previously existing ones. And the it did that again. And again. And again. It actually never stopped building new replicas and deleting the old ones.

I've seen that behavior already a few times. What helped in that situation was setting auto balance to disabled. In that case, it will finish the cycle that it's currently in, then stop

To Reproduce

See above.

Expected behavior

Even though one of the replicas is on an unschedulable node, I'd expect longhorn to realize that it already has achieved the best balance regarding fault-tolerance. I would definitely not expect it to continue recreating and deleting replicas forever.

Log or Support bundle

n/a

Environment

• Longhorn version: 1.3.1
• Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Helm
• Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: k3s v1.24.4
  • Number of management node in the cluster: 3
  • Number of worker node in the cluster: 11
• Node config
  • OS type and version: openSUSE MicroOS
  • CPU per node: 4 cores
  • Memory per node: 4-12 MB
  • Disk type(e.g. SSD/NVMe): SSD
  • Network bandwidth between the nodes: 8x10Gbe, 3x1Gbe
• Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): KVM & Baremetal
• Number of Longhorn volumes in the cluster: 37

Additional context

n/a

[innobead]

cc @c3y1huang

[c3y1huang]

Thanks for reporting. We will look into this.

[withinboredom]

Looks like the same thing (or maybe the opposite?) happens if set to best-effort and it is auto-upgrading the image and one of the nodes goes away during the upgrade.

replicas are created/destroyed in an infinite loop.

[c3y1huang]

Auto-balance best-effort will first go through the logic to achieve the balance for least-effect. So the infinite loop goes for both cases. We need to fix this bug so the setting recognizes replicas already on an un-schedulable node.

However, I am not expecting it to auto-upgrading the image, can you give some more info about the behavior?

[withinboredom]

I had the "auto-upgrade engine" turned on, and during the upgrade to 1.3.1 from 1.3.0, one of the nodes turned off while upgrading some engines. The volume then went into an infinite loop of creating replicas and deleting them until the node returned online sometime later.

I came here to report the issue and saw this here.

There were only three nodes, and each volume had three replicas (then two nodes and three replicas). It is hard to tell, but it may not be the same issue but the same behavior.

[docbobo]

I am seeing something similar just when using nodeSelector. When the nodeSelector only matches nodes in two different regions, but the strategy is set of least-effort (and maybe also best-effort) then longhorn will continuously rebuild.

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

  • test(replica-auto-balance): unschedulable node longhorn-tests#1124
  • [BUG] Continuously rebuild when auto-balance==least-effort and existing node becomes unschedulable #4502 (comment)

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is at:

  • [BUG] Continuously rebuild when auto-balance==least-effort and existing node becomes unschedulable #4502 (comment)

• Does the PR include the explanation for the fix or the feature?

• Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
  The PR for the YAML change is at:
  The PR for the chart change is at:

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at

  • fix(replica-auto-balance): unschedulable node rebuilding loop longhorn-manager#1501
  • [BACKPORT][v1.3.2] fix(replica-auto-balance): rebuilding loop on unschedulable node longhorn-manager#1506
  • [BACKPORT][v1.2.6] fix(replica-auto-balance): rebuilding loop on unschedulable node longhorn-manager#1507

• Which areas/issues this PR might have potential impacts on?
  Area manager
  Issues

• If labeled: require/LEP Has the Longhorn Enhancement Proposal PR submitted?
  The LEP PR is at

• If labeled: area/ui Has the UI issue filed or ready to be merged (including backport-needed/*)?
  The UI issue/PR is at

• If labeled: require/doc Has the necessary document PR submitted or merged (including backport-needed/*)?
  The documentation issue/PR is at

• If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue (including backport-needed/*)
  The automation skeleton/test case PR is at test(replica-auto-balance): unschedulable node longhorn-tests#1124
  The issue of automation test case implementation is at (please create by the template)

• If labeled: require/automation-engine Has the engine integration test been merged (including backport-needed/*)?
  The engine automation PR is at

• If labeled: require/manual-test-plan Has the manual test plan been documented?
  The updated manual test plan is at

• If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
  The compatibility issue is filed at

[yangchiu]

Verified passed on master-head (longhorn-manager aa79220) by executing the test_replica_auto_balance_when_replica_on_unschedulable_node automated test case, and manually running the test steps following #4502 (comment), the unexpected loop of replica deleting and recreating isn't observed.