You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
I setup another instance of seaweed on a quite busy server. Got in ~1466875 files ~60GB in size over ~10 hours via filer.copy into seaweed and then the following error happened.
copy file error: Failed to assign from [seaweed-master:9333]: assign volume failure count:1 collection:"kemono-thumbs" replication:"000" path:"/buckets/thumbs/36432417/": assign volume: failed to parse master : server port parse error: server should have hostname:port format:
That error times 8, but for different folders.
Sadly I lost all the master logs, but it seems that for a short moment the master was not reachable and caused the filer.copy to abort.
System Setup
See docker-compose further below
Ubuntu 20.04.2 LTS
output of version 8000GB 2.43 c48ef78 linux amd64
default filer.toml
[master.maintenance]
sleep_minutes = 17# sleep minutes between each script execution
[master.filer]
default = "localhost:8888"# used by maintenance scripts if the scripts needs to use fs related commands
[master.sequencer]
sequencer_etcd_urls = "http://127.0.0.1:2379"
[master.volume_growth]
copy_1 = 1# create 1 x 7 = 7 actual volumescopy_2 = 1# create 2 x 6 = 12 actual volumescopy_3 = 1# create 3 x 3 = 9 actual volumescopy_other = 1# create n x 1 = n actual volumes
[master.replication]
treat_replication_as_minimums = false
Expected behavior
Make filer.copy a bit more resilient to network errors by waiting and retrying the connection. It's not like the inserts failed, but simply the master was not reachable.
There were 8 lines of copy file error: for different folders/files and all of them have assign volume failure count:1.
There is another problem that stems from how filer.copy works, which makes this problem such a pain. Filer.copy does not know how to check if file exists and compare time and size.
Which maybe allow it to skip the file uploads that already exist in the filer. Right now, if you run filer.copy 2 times it will simply delete existing file in the filer and re-upload the "new" file again.
Additional context
The system has IO problems. Better to say, the IO is sometimes squeezed dry by webserver and other tools, which is why we want to see how seaweed fares on such a system, hosting the thumbs for now.
The text was updated successfully, but these errors were encountered:
Describe the bug
I setup another instance of seaweed on a quite busy server. Got in ~1466875 files ~60GB in size over ~10 hours via filer.copy into seaweed and then the following error happened.
That error times 8, but for different folders.
Sadly I lost all the master logs, but it seems that for a short moment the master was not reachable and caused the filer.copy to abort.
System Setup
Ubuntu 20.04.2 LTS
version 8000GB 2.43 c48ef78 linux amd64
Expected behavior
Make filer.copy a bit more resilient to network errors by waiting and retrying the connection. It's not like the inserts failed, but simply the master was not reachable.
There were 8 lines of
copy file error:
for different folders/files and all of them haveassign volume failure count:1
.There is another problem that stems from how filer.copy works, which makes this problem such a pain. Filer.copy does not know how to check if file exists and compare time and size.
Which maybe allow it to skip the file uploads that already exist in the filer. Right now, if you run filer.copy 2 times it will simply delete existing file in the filer and re-upload the "new" file again.
Additional context
The system has IO problems. Better to say, the IO is sometimes squeezed dry by webserver and other tools, which is why we want to see how seaweed fares on such a system, hosting the thumbs for now.
The text was updated successfully, but these errors were encountered: