[store] failed to write to local disk: invalid argument

[kmlebedev]

Describe the bug

logs

upload_content.go:109] uploading to http://fast-volume-6.s3-fast-volume.service.dcdp.consul:8080/825,01366342ecebf5fc: unmarshalled error http://fast-volume-6.s3-fast-volume.service.dcdp.consul:8080/825,01366342ecebf5fc: failed to write to local disk: invalid argument | fast-api-658d59f67b-vd9ct
store_replicate.go:48] failed to write to local disk: invalid argument | fast-volume-6
common.go:69] response method:POST URL:/825,01366342ecebf5fc with httpStatus:500 and JSON:{"name":"0006.part","size":5242880,"error":"failed to write to local disk: invalid argument","eTag":"971972b3"} | fast-volume-6

System Setup

2.88

[kmlebedev]

@chrislusf I found where the error is printed https://github.com/chrislusf/seaweedfs/blob/433fde4b186a8244aabeb8659bb901a89b56e213/weed/topology/store_replicate.go#L49
but I did not find where the error "invalid argument" is returned from

[chrislusf]

Is the write request using fsync?

[kmlebedev]

Is the write request using fsync?

No

W0208 09:59:31     1 upload_content.go:109] uploading to http://fast-volume-4.s3-fast-volume.service.dcdp.consul:8080/306,013669791416dd6d: unmarshalled error http://fast-volume-4.s3-fast-volume.service.dcdp.consul:8080/306,013669791416dd6d: failed to write to replicas for volume 306: [fast-volume-6.s3-fast-volume.service.dcix.consul:8080]: unmarshalled error http://fast-volume-6.s3-fast-volume.service.dcix.consul:8080/306,013669791416dd6d?ts=1644314370&ttl=&type=replicate: failed to write to local disk: invalid argument

[chrislusf]

does it happen often? can not tell, unless it is reproducible.

[kmlebedev]

does it happen often? can not tell, unless it is reproducible.

found 963 bugs in 5 days
and all about one volume 306(registry) and often about the same file

now I'm trying to find the filename by id

[kmlebedev]

@chrislusf How to enable back synchronous writing in all replicas?

Because I see other errors

failed to write to local disk: volume 1185 is read only
replicating opetations [1] is less than volume 931 replication copy count [2]

But the client s3 receives from 200 that everything is recorded and does not re-request

[famosss]

I also encountered this problem,how did you solve the problem?

[kmlebedev]

Next #2658

[kmlebedev]

I also encountered this problem,how did you solve the problem?

Now this problem should not interfere with data replication.
And I understand the reason here #2658

[kmlebedev]

Shot together with #2757

3.14

upload_content.go:114] uploading to http://fast-volume-7:8080/1827,05f983b09f8aba26?ts=1658301141&ttl=&type=replicate: unmarshalled error http://fast-volume-7:8080/1827,05f983b09f8aba26?ts=1658301141&ttl=&type=replicate: failed to write to local disk: invalid argument
store_replicate.go:50] failed to write to local disk: invalid argument

[NyaMisty]

Still occurs in version 3.64