Can't download file after uploading to cloud using volume.tier.upload

[imrebuild]

Sponsors SeaweedFS via Patreon https://www.patreon.com/seaweedfs
Report issues here. Ask questions here https://stackoverflow.com/questions/tagged/seaweedfs
Please ask questions in https://github.com/seaweedfs/seaweedfs/discussions

example of a good issue report:
#1005
example of a bad issue report:
#1008

Describe the bug
After uploading to cloud, downloading doesn't work with following error:

E0904 15:50:26.718718 common.go:287 processRangeRequest headers: map[Accept-Ranges:[bytes] Content-Disposition:[inline; filename="0001.part"] Content-Length:[4194304] Content-Type:[application/x-gzip] Etag:["f8a646e5"] Last-Modified:[Su
n, 04 Sep 2022 15:49:01 GMT] Server:[SeaweedFS Volume 30GB 3.24]] err: ReadNeedleData checksum 3705520650 expected 4171646693                                                                                                               
I0904 15:50:26.719286 filechunk_manifest.go:191 read http://100.81.110.3:8088/8,0b6a8c4d75 failed, err: http://100.81.110.3:8088/8,0b6a8c4d75?readDeleted=true: 500 Internal Server Error                                                   
I0904 15:50:26.719332 filechunk_manifest.go:197 retry reading in 1s                                                                                                                                                                         
E0904 15:50:32.799308 common.go:287 processRangeRequest headers: map[Accept-Ranges:[bytes] Content-Disposition:[inline; filename="0001.part"] Content-Length:[4194304] Content-Type:[application/x-gzip] Etag:["f8a646e5"] Last-Modified:[Su
n, 04 Sep 2022 15:49:01 GMT] Server:[SeaweedFS Volume 30GB 3.24]] err: ReadNeedleData checksum 3732444991 expected 4171646693                                                                                                               
I0904 15:50:32.799649 filechunk_manifest.go:191 read http://100.81.110.3:8088/8,0b6a8c4d75 failed, err: http://100.81.110.3:8088/8,0b6a8c4d75?readDeleted=true: 500 Internal Server Error                                                   
I0904 15:50:32.799682 filechunk_manifest.go:197 retry reading in 1.5s                                                                                                                                                                       
E0904 15:50:37.393037 common.go:287 processRangeRequest headers: map[Accept-Ranges:[bytes] Content-Disposition:[inline; filename="0001.part"] Content-Length:[4194304] Content-Type:[application/x-gzip] Etag:["f8a646e5"] Last-Modified:[Su
n, 04 Sep 2022 15:49:01 GMT] Server:[SeaweedFS Volume 30GB 3.24]] err: ReadNeedleData checksum 2990683666 expected 4171646693                                                                                                               
I0904 15:50:37.393340 filechunk_manifest.go:191 read http://100.81.110.3:8088/8,0b6a8c4d75 failed, err: http://100.81.110.3:8088/8,0b6a8c4d75?readDeleted=true: 500 Internal Server Error                                                   
I0904 15:50:37.393371 filechunk_manifest.go:197 retry reading in 2.25s                                                                                                                                                                      
E0904 15:50:42.356855 common.go:287 processRangeRequest headers: map[Accept-Ranges:[bytes] Content-Disposition:[inline; filename="0001.part"] Content-Length:[4194304] Content-Type:[application/x-gzip] Etag:["f8a646e5"] Last-Modified:[Su
n, 04 Sep 2022 15:49:01 GMT] Server:[SeaweedFS Volume 30GB 3.24]] err: ReadNeedleData checksum 2458569092 expected 4171646693                                                                                                               
I0904 15:50:42.357164 filechunk_manifest.go:191 read http://100.81.110.3:8088/8,0b6a8c4d75 failed, err: http://100.81.110.3:8088/8,0b6a8c4d75?readDeleted=true: 500 Internal Server Error                                                   
I0904 15:50:42.357187 filechunk_manifest.go:197 retry reading in 3.375s                                                                                                                                                                     
E0904 15:50:49.102094 common.go:287 processRangeRequest headers: map[Accept-Ranges:[bytes] Content-Disposition:[inline; filename="0001.part"] Content-Length:[4194304] Content-Type:[application/x-gzip] Etag:["f8a646e5"] Last-Modified:[Su
n, 04 Sep 2022 15:49:01 GMT] Server:[SeaweedFS Volume 30GB 3.24]] err: ReadNeedleData checksum 3965520242 expected 4171646693                                                                                                               
I0904 15:50:49.102431 filechunk_manifest.go:191 read http://100.81.110.3:8088/8,0b6a8c4d75 failed, err: http://100.81.110.3:8088/8,0b6a8c4d75?readDeleted=true: 500 Internal Server Error
I0904 15:50:49.102476 filechunk_manifest.go:197 retry reading in 5.0625s
E0904 15:50:54.165102 filer_server_handlers_read.go:249 failed to stream content /buckets/ttt/linux_amd64_full.tar.gz: read chunk: http://100.81.110.3:8088/8,0b6a8c4d75?readDeleted=true: 500 Internal Server Error
E0904 15:50:54.165179 common.go:287 processRangeRequest headers: map[Accept-Ranges:[bytes] Access-Control-Expose-Headers:[Content-Disposition] Content-Disposition:[inline; filename="linux_amd64_full.tar.gz"] Content-Length:[45489045] Co
ntent-Type:[application/gzip] Etag:["2031da5c6a9f273b715c05215174b0e0-11"] Last-Modified:[Sun, 04 Sep 2022 15:49:01 GMT] Server:[SeaweedFS Filer 30GB 3.24]] err: read chunk: http://100.81.110.3:8088/8,0b6a8c4d75?readDeleted=true: 500 In
ternal Server Error
2022/09/04 15:50:54 http: superfluous response.WriteHeader call from github.com/seaweedfs/seaweedfs/weed/server.processRangeRequest (common.go:288)

The upload function should work as volume.list shows the volume status correctly
volume id:8 size:12583104 collection:"ttt" file_count:3 read_only:true version:3 modified_at_second:1662306590 remote_storage_name:"s3.default" remote_storage_key:"227b5757-0719-4653-98fc-bf1a794bcf1c"

I can also download the volume using volume.tier.download.

System Setup

• List the command line to start "weed master", "weed volume", "weed filer", "weed s3", "weed mount".
  • ./weed server -s3 -dir=./data -master.volumeSizeLimitMB 100 -volume.max 0 -ip 100.81.110.3 -volume.port 8088
  • Only [storage.backend.s3.default] is changed after ./weed scaffold -config master > master.toml
• OS version: Linux
• output of weed version: version 30GB 3.24 b7a887f linux amd64
• if using filer, show the content of filer.toml

Expected behavior
File should be downloaded.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context

• File is uploaded to Seaweed through S3 API using mc (minio client).
• Both local minio and Cloudflare R2 have same issue.
• Uploading one volume or all volumes of collection have same issue.
• After downloading volume from cloud using volume.tier.download, file can be downloaded without issue.

[chrislusf]

Seems working with a remote SeaweedFS S3?

[imrebuild]

Still gettting same error

E0905 11:16:23.870976 common.go:287 processRangeRequest headers: map[Accept-Ranges:[bytes] Content-Disposition:[inline; filename="0001.part"] Content-Length:[4194304] Content-Type:[application/x-partial-download] Etag:["bc5b0170"] Last-Modified:[Mon, 05 Sep 2022 03:14:48 GMT] Server:[SeaweedFS Volume 30GB 3.24]] err: ReadNeedleData checksum 1437932273 expected 3160080752
I0905 11:16:23.871305 filechunk_manifest.go:191 read http://127.0.0.1:8081/6,03ff5651d2 failed, err: http://127.0.0.1:8081/6,03ff5651d2?readDeleted=true: 500 Internal Server Error
I0905 11:16:23.871323 filechunk_manifest.go:197 retry reading in 1s
E0905 11:16:24.889423 common.go:287 processRangeRequest headers: map[Accept-Ranges:[bytes] Content-Disposition:[inline; filename="0001.part"] Content-Length:[4194304] Content-Type:[application/x-partial-download] Etag:["bc5b0170"] Last-Modified:[Mon, 05 Sep 2022 03:14:48 GMT] Server:[SeaweedFS Volume 30GB 3.24]] err: ReadNeedleData checksum 2571093550 expected 3160080752
2022/09/05 11:16:24 http: superfluous response.WriteHeader call from github.com/seaweedfs/seaweedfs/weed/server.processRangeRequest (common.go:288)
I0905 11:16:24.889694 filechunk_manifest.go:191 read http://127.0.0.1:8081/6,03ff5651d2 failed, err: unexpected EOF
I0905 11:16:24.889717 filechunk_manifest.go:197 retry reading in 1.5s
E0905 11:16:26.406323 common.go:287 processRangeRequest headers: map[Accept-Ranges:[bytes] Content-Disposition:[inline; filename="0001.part"] Content-Length:[4194304] Content-Type:[application/x-partial-download] Etag:["bc5b0170"] Last-Modified:[Mon, 05 Sep 2022 03:14:48 GMT] Server:[SeaweedFS Volume 30GB 3.24]] err: ReadNeedleData checksum 666268485 expected 3160080752
2022/09/05 11:16:26 http: superfluous response.WriteHeader call from github.com/seaweedfs/seaweedfs/weed/server.processRangeRequest (common.go:288)
I0905 11:16:26.406561 filechunk_manifest.go:191 read http://127.0.0.1:8081/6,03ff5651d2 failed, err: unexpected EOF
I0905 11:16:26.406576 filechunk_manifest.go:197 retry reading in 2.25s
E0905 11:16:28.672993 common.go:287 processRangeRequest headers: map[Accept-Ranges:[bytes] Content-Disposition:[inline; filename="0001.part"] Content-Length:[4194304] Content-Type:[application/x-partial-download] Etag:["bc5b0170"] Last-Modified:[Mon, 05 Sep 2022 03:14:48 GMT] Server:[SeaweedFS Volume 30GB 3.24]] err: ReadNeedleData checksum 320167853 expected 3160080752
2022/09/05 11:16:28 http: superfluous response.WriteHeader call from github.com/seaweedfs/seaweedfs/weed/server.processRangeRequest (common.go:288)
I0905 11:16:28.673280 filechunk_manifest.go:191 read http://127.0.0.1:8081/6,03ff5651d2 failed, err: unexpected EOF
I0905 11:16:28.673301 filechunk_manifest.go:197 retry reading in 3.375s
E0905 11:16:32.064102 common.go:287 processRangeRequest headers: map[Accept-Ranges:[bytes] Content-Disposition:[inline; filename="0001.part"] Content-Length:[4194304] Content-Type:[application/x-partial-download] Etag:["bc5b0170"] Last-Modified:[Mon, 05 Sep 2022 03:14:48 GMT] Server:[SeaweedFS Volume 30GB 3.24]] err: ReadNeedleData checksum 2192220661 expected 3160080752
I0905 11:16:32.064479 filechunk_manifest.go:191 read http://127.0.0.1:8081/6,03ff5651d2 failed, err: http://127.0.0.1:8081/6,03ff5651d2?readDeleted=true: 500 Internal Server Error
I0905 11:16:32.064502 filechunk_manifest.go:197 retry reading in 5.0625s
E0905 11:16:37.127571 filer_server_handlers_read.go:249 failed to stream content /buckets/tttt/weed: read chunk: http://127.0.0.1:8081/6,03ff5651d2?readDeleted=true: 500 Internal Server Error
E0905 11:16:37.127605 common.go:287 processRangeRequest headers: map[Accept-Ranges:[bytes] Access-Control-Expose-Headers:[Content-Disposition] Content-Disposition:[inline; filename="weed"] Content-Length:[108227095] Content-Type:[application/x-executable] Etag:["f024fd5e2c1db2adc2f1fd9ea6915d1d-26"] Last-Modified:[Mon, 05 Sep 2022 03:14:48 GMT] Server:[SeaweedFS Filer 30GB 3.24]] err: read chunk: http://127.0.0.1:8081/6,03ff5651d2?readDeleted=true: 500 Internal Server Error
2022/09/05 11:16:37 http: superfluous response.WriteHeader call from github.com/seaweedfs/seaweedfs/weed/server.processRangeRequest (common.go:288)

Setup remote SeaweedFS S3

Start first SeaweedFS S3: ./weed server -s3 -dir=./data1 -master.volumeSizeLimitMB 100 -volume.max 0 -ip 127.0.0.1
Confirgure SeaweedFS S3: s3.configure -access_key=anyanyany -secret_key=anyanyany -user=me -actions=Read,Write,List,Tagging,Admin -apply
Set alias on mc: mc alias set weed http://localhost:8333 anyanyany anyanyany
Create bucket: mc mb weed/ttt

Setup local SeaweedFS S3

Modify master.toml:

[storage.backend.s3.default]
enabled = true
aws_access_key_id = "anyanyany"
aws_secret_access_key = "anyanyany"
region = "us-east-1"
bucket = "ttt"
endpoint = "http://localhost:8333"
storage_class = "STANDARD"

Start second SeaweedFS S3: ./weed server -s3 -dir=./data2 -master.volumeSizeLimitMB 100 -volume.max 0 -ip 127.0.0.1 -filer.port 8887 -master.port 9334 -s3.port 8334 -volume.port 8081
Set alias on mc: mc alias set weed1 http://localhost:8334
Create bucket: mc mb weed1/tttt

Testing

Upload file through S3: mc cp ./weed weed1/tttt/
Enter weed shell for local: ./weed shell -master localhost:9334
Check volume status: volume.list
Lock, upload one volume, unlock then quit shell: volume.tier.upload -volumeId x -dest s3 -collection tttt
Check if volume is uploaded to remote: mc ls weed/ttt
Check if file exists on local: mc ls weed1/tttt
Download file: mc cp weed1/tttt/weed ~/ -> boom