2.04: volumeServer.evacuate moves one volume only

[onlyjob]

volumeServer.evacuate fails to move all volumes but succeeds in moving one volume at a time.

I've started a new volume server and I'm trying to move data from another volume server (which I want to decommission):

> volumeServer.evacuate -node 192.168.0.250:9334 -force
moving volume 2 192.168.0.250:9334 => 192.168.0.204:8080
2020-10-14 00:24:04.230186 I | copying volume 2 from 192.168.0.250:9334 to 192.168.0.204:8080
2020-10-14 00:24:07.278448 I | tailing volume 2 from 192.168.0.250:9334 to 192.168.0.204:8080
2020-10-14 00:24:19.473121 I | deleting volume 2 from 192.168.0.250:9334
2020-10-14 00:24:19.682220 I | moved volume 2 from 192.168.0.250:9334 to 192.168.0.204:8080
error: failed to move volume 19 from 192.168.0.250:9334

> volumeServer.evacuate -node 192.168.0.250:9334 -force
moving volume 4 192.168.0.250:9334 => 192.168.0.204:8080
2020-10-14 00:24:30.661134 I | copying volume 4 from 192.168.0.250:9334 to 192.168.0.204:8080
2020-10-14 00:24:32.954465 I | tailing volume 4 from 192.168.0.250:9334 to 192.168.0.204:8080
2020-10-14 00:24:45.109108 I | deleting volume 4 from 192.168.0.250:9334
2020-10-14 00:24:45.272594 I | moved volume 4 from 192.168.0.250:9334 to 192.168.0.204:8080
error: failed to move volume 3 from 192.168.0.250:9334

Re-trying the command volumeServer.evacuate -node 192.168.0.250:9334 -force always stops after processing one volume which it manages to evacuate successfully. With one volume processed per command invocation, running volumeServer.evacuate enough times eventually managed to move/delete all the volumes from server 192.168.0.250.

[chrislusf]

I could not reproduce this locally. It would be good to see the output of volume.list before a volumeServer.evacuate execution.

[PeterCxy]

I can confirm that this is the case in my deployment -- it always stops after moving one volume stating it "failed to move" the next volume.

[PeterCxy]

one thing to note though, is that my setup consists of servers of different sizes, so no two servers have the same number of available volumes. This seems to cause issues with volume.balance, and apparently volumeServer.evacuate shares some routines with volume.balance in terms of selecting the new best server to move to.

[PeterCxy]

@chrislusf Could you test the case where every volume server has a different number of available volumes and see if this can be reproduced? Also volume.balance does not seem to behave correctly in this case.

[chrislusf]

Please share the output of volume.list

[PeterCxy]

@chrislusf I will DM you (on Patreon) with the link to my volume list because 1) it's huge; 2) it contains some of my S3 bucket names that I may not want to make public.