[volume.balance] gracefull move volumes during active recording

[kmlebedev]

Describe the bug

In a regulat task runs

lock
 volume.check.disk -v -force
 volume.fix.replication
 volume.balance -collection EACH_COLLECTION  -force
volume.check.disk -v -force
 unlock | weed shell

logs

> > load collection xxxxxxx-store volume 110 index size 42080 from fast-volume-0.s3-fast-volume.dc2:8080 ...
load collection xxxxxxx-store volume 110 index size 42080 from fast-volume-2.s3-fast-volume.dc1:8080 ...
volume 110 fast-volume-0.s3-fast-volume.dc2:8080 has 2630 entries, fast-volume-2.s3-fast-volume.dc1:8080 missed 0 entries
volume 110 fast-volume-2.s3-fast-volume.dc1:8080 has 2630 entries, fast-volume-0.s3-fast-volume.dc2:8080 missed 0 entries
load collection xxxxxxx-store volume 111 index size 43376 from fast-volume-3.s3-fast-volume.dc1:8080 ...
load collection xxxxxxx-store volume 111 index size 43376 from fast-volume-2.s3-fast-volume.dc2:8080 ...
volume 111 fast-volume-3.s3-fast-volume.dc1:8080 has 2711 entries, fast-volume-2.s3-fast-volume.dc2:8080 missed 0 entries
volume 111 fast-volume-2.s3-fast-volume.dc2:8080 has 2711 entries, fast-volume-3.s3-fast-volume.dc1:8080 missed 0 entries
load collection xxxxxxx-store volume 113 index size 1632 from fast-volume-1.s3-fast-volume.dc1:8080 ...
load collection xxxxxxx-store volume 113 index size 1632 from fast-volume-1.s3-fast-volume.dc2:8080 ...
volume 113 fast-volume-1.s3-fast-volume.dc1:8080 has 102 entries, fast-volume-1.s3-fast-volume.dc2:8080 missed 0 entries
volume 113 fast-volume-1.s3-fast-volume.dc2:8080 has 102 entries, fast-volume-1.s3-fast-volume.dc1:8080 missed 0 entries
> >   moving  volume xxxxxxx-store_112 fast-volume-1.s3-fast-volume.dc2:8080 => fast-volume-0.s3-fast-volume.dc2:8080
2021/07/13 09:26:11 copying volume 112 from fast-volume-1.s3-fast-volume.dc2:8080 to fast-volume-0.s3-fast-volume.dc2:8080
I0713 09:26:11     8 masterclient.go:139] adminShell: fast-volume-0.s3-fast-volume.dc2:8080 masterClient adds volume 112
2021/07/13 09:26:11 tailing volume 112 from fast-volume-1.s3-fast-volume.dc2:8080 to fast-volume-0.s3-fast-volume.dc2:8080
2021/07/13 09:26:23 deleting volume 112 from fast-volume-1.s3-fast-volume.dc2:8080
2021/07/13 09:26:23 moved volume 112 from fast-volume-1.s3-fast-volume.dc2:8080 to fast-volume-0.s3-fast-volume.dc2:8080
2021/07/13 09:26:23 copying volume 113 from fast-volume-1.s3-fast-volume.dc1:8080 to fast-volume-3.s3-fast-volume.dc1:8080
  moving  volume xxxxxxx-store_113 fast-volume-1.s3-fast-volume.dc1:8080 => fast-volume-3.s3-fast-volume.dc1:8080
I0713 09:26:23     8 masterclient.go:143] adminShell: fast-volume-1.s3-fast-volume.dc2:8080 masterClient removes volume 112
I0713 09:26:23     8 masterclient.go:139] adminShell: fast-volume-3.s3-fast-volume.dc1:8080 masterClient adds volume 113
2021/07/13 09:26:23 tailing volume 113 from fast-volume-1.s3-fast-volume.dc1:8080 to fast-volume-3.s3-fast-volume.dc1:8080
2021/07/13 09:26:35 deleting volume 113 from fast-volume-1.s3-fast-volume.dc1:8080
2021/07/13 09:26:35 moved volume 113 from fast-volume-1.s3-fast-volume.dc1:8080 to fast-volume-3.s3-fast-volume.dc1:8080
I0713 09:26:35     8 masterclient.go:143] adminShell: fast-volume-1.s3-fast-volume.dc1:8080 masterClient removes volume 113
> load collection xxxxxxx-store volume 110 index size 42384 from fast-volume-0.s3-fast-volume.dc2:8080 ...
load collection xxxxxxx-store volume 110 index size 42384 from fast-volume-2.s3-fast-volume.dc1:8080 ...
volume 110 fast-volume-0.s3-fast-volume.dc2:8080 has 2649 entries, fast-volume-2.s3-fast-volume.dc1:8080 missed 0 entries
volume 110 fast-volume-2.s3-fast-volume.dc1:8080 has 2649 entries, fast-volume-0.s3-fast-volume.dc2:8080 missed 0 entries
load collection xxxxxxx-store volume 112 index size 1504 from fast-volume-0.s3-fast-volume.dc1:8080 ...
load collection xxxxxxx-store volume 112 index size 1408 from fast-volume-0.s3-fast-volume.dc2:8080 ...
volume 112 fast-volume-0.s3-fast-volume.dc1:8080 has 94 entries, fast-volume-0.s3-fast-volume.dc2:8080 missed 6 entries
read 112,3539333463 fast-volume-0.s3-fast-volume.dc1:8080 => fast-volume-0.s3-fast-volume.dc2:8080
read 112,3539333464 fast-volume-0.s3-fast-volume.dc1:8080 => fast-volume-0.s3-fast-volume.dc2:8080
read 112,3539333630 fast-volume-0.s3-fast-volume.dc1:8080 => fast-volume-0.s3-fast-volume.dc2:8080
read 112,3539333634 fast-volume-0.s3-fast-volume.dc1:8080 => fast-volume-0.s3-fast-volume.dc2:8080
read 112,3539333636 fast-volume-0.s3-fast-volume.dc1:8080 => fast-volume-0.s3-fast-volume.dc2:8080
read 112,3539333665 fast-volume-0.s3-fast-volume.dc1:8080 => fast-volume-0.s3-fast-volume.dc2:8080
volume 112 fast-volume-0.s3-fast-volume.dc2:8080 has 88 entries, fast-volume-0.s3-fast-volume.dc1:8080 missed 0 entries
load collection xxxxxxx-store volume 109 index size 43280 from fast-volume-1.s3-fast-volume.dc1:8080 ...
load collection xxxxxxx-store volume 109 index size 43280 from fast-volume-1.s3-fast-volume.dc2:8080 ...
volume 109 fast-volume-1.s3-fast-volume.dc1:8080 has 2705 entries, fast-volume-1.s3-fast-volume.dc2:8080 missed 0 entries
volume 109 fast-volume-1.s3-fast-volume.dc2:8080 has 2705 entries, fast-volume-1.s3-fast-volume.dc1:8080 missed 0 entries
load collection xxxxxxx-store volume 111 index size 43760 from fast-volume-3.s3-fast-volume.dc1:8080 ...
load collection xxxxxxx-store volume 111 index size 43760 from fast-volume-2.s3-fast-volume.dc2:8080 ...
volume 111 fast-volume-3.s3-fast-volume.dc1:8080 has 2735 entries, fast-volume-2.s3-fast-volume.dc2:8080 missed 0 entries
volume 111 fast-volume-2.s3-fast-volume.dc2:8080 has 2735 entries, fast-volume-3.s3-fast-volume.dc1:8080 missed 0 entries

then catch errors

E0713 09:27:42     1 filer_server_handlers_read.go:168] failed to stream content /buckets/xxxxx-store/files/2021/07/13/117b4ac7-c197-430b-abf4-c6046a51a0ec: read chunk: http://fast-volume-0.s3-fast-volume.dc2:8080/112,0593aec3661828?readDeleted=true: 404 Not Found
E0713 09:27:53     1 filer_server_handlers_read.go:168] failed to stream content /buckets/xxxxx-store/files/2021/07/13/117b4ac7-c197-430b-abf4-c6046a51a0ec: read chunk: http://fast-volume-0.s3-fast-volume.dc2:8080/112,0593aec3661828?readDeleted=true: 404 Not Found

System Setup

2.53

Expected behavior
moving without read chunk errors

[chrislusf]

Added code to ensure the volumes are in sync.