S3 upload failed for filenames including '%' character

[AlexMikhalev]

Describe the bug
S3 filer API can't handle files with '%' sign despite it being the valid name (and valid object for s3 and minio, I found plenty of those inside Photo library on Mac):

touch %%test.file.txt 
touch %%test.file.txt
aws --endpoint-url http://rpi4node3:8333/ s3 cp %test.file.txt s3://testbucket/
upload failed: ./%test.file.txt to s3://testbucket/%test.file.txt An error occurred (InternalError) when calling the PutObject operation (reached max retries: 2): We encountered an internal error, please try again.

Upload via weed upload works:

 > weed upload -master="rpi4node3:9333" %%test.file.txt
[{"fileName":"%%test.file.txt","url":"rpi4node3:8088/13,022beaf4f28a09","fid":"13,022beaf4f28a09"}]
> weed upload -master="rpi4node3:9333" %test.file.txt 
[{"fileName":"%test.file.txt","url":"rpi4node3:8088/6,022beb5bdfd3a2","fid":"6,022beb5bdfd3a2"}]

System Setup
Master started with:

~/go/bin/weed server -filer -s3 -ip=rpi4node3 -volume.max=0,0 -master.volumeSizeLimitMB=1024 -volume.port=8088 -dir=/mnt/usbslow/weed,/mnt/usbfast/weed/ -master.dir=/mnt/usbfast/weed/master

Filer config:

[redis_cluster2]
enabled = true
addresses = [
    "rpi4node1:6379",
    "rpi4node2:6379",
    "rpi4node3:6379"
]
password = ""
# allows reads from slave servers or the master, but all writes still go to the master
readOnly = false
# automatically use the closest Redis server for reads
routeByLatency = false
# This changes the data layout. Only add new directories. Removing/Updating will cause data loss.
superLargeDirectories = []

Volumes:

/usr/local/bin/weed volume -mserver="rpi4node3:9333" -max=1000,1000 -port=8088 -disk=hdd,ssd -dir=/mnt/usbslow/weed,/mnt/usbfast/weed/ &

Master weed version:

~/go/bin/weed version
version 30GB 2.29 400de380 linux arm64

Complied from:

commit 400de380f48c44c7700fdf7e8f247bf856662c10 (HEAD -> master, origin/master, origin/HEAD)
Author: Chris Lu <chris.lu@gmail.com>
Date:   Fri Mar 5 02:29:38 2021 -0800

    volume server: support tcp direct put/get/delete

Expected behavior

Successful upload of files via S3.
Additional context
You guessed seaweed fs runs on RPI4 cluster. I found it's a really amazing project and very easy to setup.

[chrislusf]

Thanks for reporting! Added a fix.

[AlexMikhalev]

Thank you, fix works for upload, remove isn't do you want separate ticket for?

aws --endpoint-url http://rpi4node3:8333/ s3 rm s3://testbucket/%test.file.txt
delete failed: s3://testbucket/%test.file.txt An error occurred (InternalError) when calling the DeleteObject operation (reached max retries: 2): We encountered an internal error, please try again

[chrislusf]

Thanks for the quick update! I added one more fix.

[AlexMikhalev]

I really appreciate how quickly you are fixing it, but I found one more:

NewRequest http://rpi4node3:8888/buckets/mac_home_alex/Library/Application Support/TextMate/Managed/Bundles/CSS.tmbundle/Snippets/Fixed Position Bottom 100% wide IE6.tmSnippet: parse "http://rpi4node3:8888/buckets/mac_home_alex/Library/Application Support/TextMate/Managed/Bundles/CSS.tmbundle/Snippets/Fixed Position Bottom 100% wide IE6.tmSnippet": invalid URL escape "% w"

Offending filename: Fixed Position Bottom 100% wide IE6.tmSnippet
This is from mac mini backup.

[chrislusf]

How to reproduce it?

[AlexMikhalev]

touch 'Fixed Position Bottom 100% wide IE6.tmSnippet'
aws --endpoint-url http://rpi4node3:8333/ s3 cp Fixed\ Position\ Bottom\ 100%\ wide\ IE6.tmSnippet s3://testbucket/
upload: ./Fixed Position Bottom 100% wide IE6.tmSnippet to s3://testbucket/Fixed Position Bottom 100% wide IE6.tmSnippet

Double-checking with the latest commit 5fbcaaf seems to be working so far.