Error "too many requests" when uploading files now common

[ncw]

The rclone integration tests for storj ran smoothly until 2023-04-19 when they started failing with error "too many requests" quite regularly when uploading files.

2023/05/16 05:12:53 DEBUG : FS sj://rclone-test-jofayup4huhiviz0napopiq3: cp input ./foo # [HashesOption([])] 3
2023/05/16 05:12:53 ERROR : foo: Failed to copy: uplink: too many requests

This seems to be entirely on uploading files.

Has something changed in the backend?

This is the versions of modules I'm using

I note v1.10.0 is quite old (December 2022) but there isn't a newer release tagged. I can try on master if you think that is worth a go.

	storj.io/uplink v1.10.0
	storj.io/common v0.0.0-20221123115229-fed3e6651b63 // indirect
	storj.io/drpc v0.0.32 // indirect

Thanks

[mniewrzal]

Hi @ncw
is it possible that those tests are doing several uploads to the same location one by one? Recently we introduced a rate limit for uploads to the same path.

[ncw]

Yes that is very likely @mniewrzal

Should rclone retry the upload in this case? Perhaps after some time?

[ncw]

Any thoughts on this? This has killed the rclone + storj integration tests :-(

These are all the failures.

[mniewrzal]

Could you provide logs from some of failed tests?

[ncw]

The overview of tests is here and you can download logs from there: https://pub.rclone.org/integration-tests/current/

However this one is representative

"go test -v -timeout 1h0m0s -remote TestStorj: -verbose -test.run '^TestIntegration$/^FsMkdir$/^FsPutFiles$'" - Starting (try 5/5)
=== RUN   TestIntegration
    fstests.go:423: Using remote "TestStorj:"
2023/06/20 05:13:11 DEBUG : Creating backend with remote "TestStorj:rclone-test-dupipey8yogicap7sadajux5"
2023/06/20 05:13:11 DEBUG : Using config file from "/home/rclone/.rclone.conf"
2023/06/20 05:13:11 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: connecting...
2023/06/20 05:13:11 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: connected: <nil>
2023/06/20 05:13:11 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: mkdir -p ./
2023/06/20 05:13:11 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: ls -R ./
2023/06/20 05:13:11 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: OBJ ls -R ./ ("rclone-test-dupipey8yogicap7sadajux5", "")
=== RUN   TestIntegration/FsMkdir
2023/06/20 05:13:11 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: mkdir -p ./
2023/06/20 05:13:11 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: ls -R ./
2023/06/20 05:13:11 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: OBJ ls -R ./ ("rclone-test-dupipey8yogicap7sadajux5", "")
2023/06/20 05:13:11 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: mkdir -p ./
=== RUN   TestIntegration/FsMkdir/FsPutFiles
2023/06/20 05:13:11 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: cp input ./file name.txt # [] 100
2023/06/20 05:13:11 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: stat ./file name.txt
2023/06/20 05:13:11 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: cp input ./hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt # [] 100
2023/06/20 05:13:12 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: stat ./hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt
2023/06/20 05:13:12 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: cp input ./file name.txt # [] 100
    fstests.go:154: 
        	Error Trace:	/home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:154
        	            				/home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:170
        	            				/home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:223
        	            				/home/rclone/go/src/github.com/rclone/rclone/fstest/fstests/fstests.go:885
        	Error:      	Received unexpected error:
        	            	uplink: too many requests
        	            		storj.io/uplink.convertKnownErrors:69
        	            		storj.io/uplink.(*Upload).Commit:181
        	            		github.com/rclone/rclone/backend/storj.(*Fs).Put:582
        	            		github.com/rclone/rclone/fstest/fstests.PutTestContentsMetadata.func1:189
        	            		github.com/rclone/rclone/fstest/fstests.retry:146
        	            		github.com/rclone/rclone/fstest/fstests.PutTestContentsMetadata:170
        	            		github.com/rclone/rclone/fstest/fstests.testPutMimeType:223
        	            		github.com/rclone/rclone/fstest/fstests.Run.func13.13:885
        	            		testing.tRunner:1446
        	Test:       	TestIntegration/FsMkdir/FsPutFiles
        	Messages:   	Put
2023/06/20 05:13:12 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: ls -R ./
2023/06/20 05:13:12 DEBUG : FS sj://rclone-test-dupipey8yogicap7sadajux5: OBJ ls -R ./ ("rclone-test-dupipey8yogicap7sadajux5", "")
--- FAIL: TestIntegration (1.17s)
    --- FAIL: TestIntegration/FsMkdir (0.84s)
        --- FAIL: TestIntegration/FsMkdir/FsPutFiles (0.39s)
FAIL
exit status 1
FAIL	github.com/rclone/rclone/backend/storj	1.186s
"go test -v -timeout 1h0m0s -remote TestStorj: -verbose -test.run '^TestIntegration$/^FsMkdir$/^FsPutFiles$'" - Finished ERROR in 1.991328221s (try 5/5): exit status 1: Failed [TestIntegration/FsMkdir/FsPutFiles]

[mniewrzal]

I think it's the same limitation we talked earlier, uploading to the same location many times. I just realized that I never replied to your comment from May and I'm deeply sorry for that.

Should rclone retry the upload in this case? Perhaps after some time?

Yes, that would be the best solution.

I ran those integration tests against local environment and I had no issues with uplink: too many requests but we have different limit defaults for prod and test environments. On production limit is one upload to the same location per 1s so if retry will be needed it needs to be at lest 1s.

[ncw]

That was very helpful info thank you. I can make a workaround for that which will sleep for 1 second then retry.

I think this is probably a problem which only affects rclone's integration tests and not actual users unless they are in the habit of altering the same file very frequently.

I'll close this now.

[mniewrzal]

Thank you for looking into this. Let me know if I will be able to help with anything else.