filer returns error 500 when pushing files while rebalance in progress

[bmillemathias]

Describe the bug
I made a script to test http access over filer where I upload constantly small files using http.
I use the python-weed library that I've slightly modified to work with python 3
during the upload, if I launch a rebalance I get an HTTP error 500.

Steps

• upload in progress of small files (up to 10k)
• rebalance data

> volume.balance -force                                                            
moving volume small-files_36 xxx.xxx.xxx.222:8080 => xxx.xxx.xxx.28:8080              
2020/12/24 14:28:15 copying volume 36 from xxx.xxx.xxx.222:8080 to xxx.xxx.xxx.28:8080
2020/12/24 14:28:18 tailing volume 36 from xxx.xxx.xxx.222:8080 to xxx.xxx.xxx.28:8080
2020/12/24 14:28:30 deleting volume 36 from xxx.xxx.xxx.222:8080                   
2020/12/24 14:28:30 moved volume 36 from xxx.xxx.xxx.222:8080 to xxx.xxx.xxx.28:8080  

result: get error 500 on some file

b9746a231da549c5801fdf0a6d66be46
ecf5bef697a8424488976d54002ae7b2
500 POST http://xxx.xxx.xxx.222:8889/buckets/small-files/ecf5bef697a8424488976d54002ae7b2
16db7e72ed6c452396326107ce423859
0d3e0f2eb5834323b6ff3340d08a2a79
9b79215d717647ffa2db7d935753bcbf
aeb4fe02749445109652ce06ac1cdc73
2f11ebac99bb416ebf4cc5e73678b69e
5a95d620503d47bab88fd7238e357c4f
500 POST http://.222:8889/buckets/small-files/5a95d620503d47bab88fd7238e357c4f
c2d4b9c524eb4d9b86b11794a7d3d120
500 POST http://xxx.xxx.xxx.222:8889/buckets/small-files/c2d4b9c524eb4d9b86b11794a7d3d120
c7a58b0f1bc6463ca75c7b00ccc5f450
aaf9219a090140ecad9ba95d0a55b718
500 POST http://xxx.xxx.xxx.222:8889/buckets/small-files/aaf9219a090140ecad9ba95d0a55b718
0e63e49ff6134dc7ba33456ec73a9a15
500 POST http://xxx.xxx.xxx.222:8889/buckets/small-files/0e63e49ff6134dc7ba33456ec73a9a15

System Setup

• List the command line to start "weed master", "weed volume", "weed filer", "weed s3", "weed mount".
• OS version: ubuntu 18.04
• output of weed version: 2.16
• if using filer, show the content of filer.toml: no content

Additional context
The cluster is set as follow

• one master
• four volume
• one filer (hosted on the same node than the master)

[chrislusf]

Added some retry. It is better to also try to retry from the client side also.

[bmillemathias]

Added some retry. It is better to also try to retry from the client side also.

Understood, I tried to find docs about such statement and found nothing.