New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rclone started returning errors after uploading at most 800gb #1591

Open
kajeagentspi opened this Issue Aug 10, 2017 · 5 comments

Comments

Projects
None yet
2 participants
@kajeagentspi

kajeagentspi commented Aug 10, 2017

When filing an issue, please include the following information if possible as well as a description of the problem. Make sure you test with the latest beta of rclone.

https://beta.rclone.org/
https://rclone.org/downloads/

If you've just got a question or aren't sure if you've found a bug then please use the rclone forum instead of filing an issue.

What is your rclone version (eg output from rclone -V)

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

rclone: Version "v1.37-028-g9e3ad0f7-treeβ
Ubuntu 17.04
Google Drive
rclone copy scripts grace:scripts
I can still download 10gb files after encountering the error so it's not like the usual 403 we encounter API limits.

2017/08/10 08:32:53 DEBUG : rclone: Version "v1.37-028-g9e3ad0f7-treeβ" starting with parameters ["rclone" "copy" "../scripts/" "grace:scripts" "-vv"]
2017/08/10 08:32:54 INFO : Google drive root 'scripts': Modify window is 1ms
2017/08/10 08:32:54 DEBUG : pacer: Rate limited, sleeping for 1.56304443s (1 consecutive low level retries)
2017/08/10 08:32:54 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)
2017/08/10 08:32:54 DEBUG : pacer: Rate limited, sleeping for 2.640240282s (2 consecutive low level retries)
2017/08/10 08:32:54 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)
2017/08/10 08:32:54 DEBUG : pacer: Rate limited, sleeping for 4.037471099s (3 consecutive low level retries)
2017/08/10 08:32:54 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)
2017/08/10 08:32:54 DEBUG : pacer: Rate limited, sleeping for 8.719298335s (4 consecutive low level retries)
2017/08/10 08:32:54 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)
2017/08/10 08:32:56 DEBUG : pacer: Resetting sleep to minimum 10ms on success
2017/08/10 08:33:05 DEBUG : pacer: Rate limited, sleeping for 1.833336691s (1 consecutive low level retries)
2017/08/10 08:33:05 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2017/08/10 08:33:05 DEBUG : pacer: Rate limited, sleeping for 2.782809181s (2 consecutive low level retries)
2017/08/10 08:33:05 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2017/08/10 08:33:05 DEBUG : pacer: Rate limited, sleeping for 4.985657727s (3 consecutive low level retries)
2017/08/10 08:33:05 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2017/08/10 08:33:07 DEBUG : pacer: Rate limited, sleeping for 8.219435813s (4 consecutive low level retries)
2017/08/10 08:33:07 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User rate limit exceeded, userRateLimitExceeded)
2017/08/10 08:33:07 DEBUG : rvlistDCF447.txt.save: Received error: googleapi: Error 403: User rate limit exceeded, userRateLimitExceeded - low level retry 1/10
2017/08/10 08:33:12 DEBUG : pacer: Resetting sleep to minimum 10ms on success
2017/08/10 08:33:20 DEBUG : pacer: Rate limited, sleeping for 1.655539466s (1 consecutive low level retries)
2017/08/10 08:33:20 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User rate limit exceeded, userRateLimitExceeded)
2017/08/10 08:33:20 DEBUG : boostvlive.sh: Received error: googleapi: Error 403: User rate limit exceeded, userRateLimitExceeded - low level retry 1/10
2017/08/10 08:33:20 DEBUG : pacer: Resetting sleep to minimum 10ms on success
2017/08/10 08:33:22 DEBUG : pacer: Rate limited, sleeping for 1.003491379s (1 consecutive low level retries)
2017/08/10 08:33:22 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User rate limit exceeded, userRateLimitExceeded)
2017/08/10 08:33:22 DEBUG : pshlistF191B3.txt: Received error: googleapi: Error 403: User rate limit exceeded, userRateLimitExceeded - low level retry 1/10
2017/08/10 08:33:22 DEBUG : pacer: Rate limited, sleeping for 2.121060521s (2 consecutive low level retries)
2017/08/10 08:33:22 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)
2017/08/10 08:33:22 DEBUG : pacer: Rate limited, sleeping for 4.080632037s (3 consecutive low level retries)
2017/08/10 08:33:22 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)
2017/08/10 08:33:22 DEBUG : rvlistDCF447.txt.save: Received error: googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded - low level retry 2/10
2017/08/10 08:33:22 DEBUG : pacer: Rate limited, sleeping for 8.780885081s (4 consecutive low level retries)
2017/08/10 08:33:22 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)
2017/08/10 08:33:22 DEBUG : boostvlive.sh: Received error: googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded - low level retry 2/10
2017/08/10 08:33:22 DEBUG : pacer: Rate limited, sleeping for 16.787762735s (5 consecutive low level retries)
2017/08/10 08:33:22 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)
2017/08/10 08:33:22 DEBUG : rvlistDCF447.txt: Received error: googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded - low level retry 1/10
2017/08/10 08:33:23 DEBUG : pacer: Resetting sleep to minimum 10ms on success
2017/08/10 08:33:40 DEBUG : pacer: Rate limited, sleeping for 1.087713558s (1 consecutive low level retries)
2017/08/10 08:33:40 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User rate limit exceeded, userRateLimitExceeded)
2017/08/10 08:33:40 DEBUG : pshlistF191B3.txt: Received error: googleapi: Error 403: User rate limit exceeded, userRateLimitExceeded - low level retry 2/10
2017/08/10 08:33:40 DEBUG : pacer: Resetting sleep to minimum 10ms on success
2017/08/10 08:33:41 DEBUG : pacer: Rate limited, sleeping for 1.736565241s (1 consecutive low level retries)
2017/08/10 08:33:41 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)
2017/08/10 08:33:41 DEBUG : rvlistDCF447.txt.save: Received error: googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded - low level retry 3/10
2017/08/10 08:33:41 DEBUG : pacer: Rate limited, sleeping for 2.447019912s (2 consecutive low level retries)
2017/08/10 08:33:41 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)
2017/08/10 08:33:41 DEBUG : boostvlive.sh: Received error: googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded - low level retry 3/10
2017/08/10 08:33:41 DEBUG : pacer: Rate limited, sleeping for 4.354758304s (3 consecutive low level retries)
2017/08/10 08:33:41 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)
2017/08/10 08:33:41 DEBUG : rvlistDCF447.txt: Received error: googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded - low level retry 2/10
2017/08/10 08:33:41 DEBUG : pacer: Rate limited, sleeping for 8.742122016s (4 consecutive low level retries)
2017/08/10 08:33:41 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)
2017/08/10 08:33:41 DEBUG : pshlistF191B3.txt: Received error: googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded - low level retry 3/10
2017/08/10 08:33:41 DEBUG : pacer: Rate limited, sleeping for 16.713120295s (5 consecutive low level retries)
2017/08/10 08:33:41 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)
^C

@ncw

This comment has been minimized.

Show comment
Hide comment
@ncw

ncw Aug 10, 2017

Owner

Those errors userRateLimitExceeded are how google drive does its rate limiting - it doesn't look like it is actually causing you a problem though.

Have you made your own client_id?

Owner

ncw commented Aug 10, 2017

Those errors userRateLimitExceeded are how google drive does its rate limiting - it doesn't look like it is actually causing you a problem though.

Have you made your own client_id?

@kajeagentspi

This comment has been minimized.

Show comment
Hide comment
@kajeagentspi

kajeagentspi Aug 10, 2017

I didn't make my own. Will try it later and report back.

kajeagentspi commented Aug 10, 2017

I didn't make my own. Will try it later and report back.

@kajeagentspi

This comment has been minimized.

Show comment
Hide comment
@kajeagentspi

kajeagentspi Aug 10, 2017

Hi apparently I'm not the only one experiencing this. Another user reported even using his own client id.

https://www.reddit.com/r/DataHoarder/comments/6sryfv/anybody_encountering_403_while_using_rclone_this/

kajeagentspi commented Aug 10, 2017

Hi apparently I'm not the only one experiencing this. Another user reported even using his own client id.

https://www.reddit.com/r/DataHoarder/comments/6sryfv/anybody_encountering_403_while_using_rclone_this/

@ncw

This comment has been minimized.

Show comment
Hide comment
@ncw

ncw Aug 10, 2017

Owner

I'm not sure I can do anything about this :-( Presumably it will go back to normal after some time?

Owner

ncw commented Aug 10, 2017

I'm not sure I can do anything about this :-( Presumably it will go back to normal after some time?

@ncw

This comment has been minimized.

Show comment
Hide comment
@ncw

ncw Aug 10, 2017

Owner

There is a thread about this on the forum which has more discussion in it.

Owner

ncw commented Aug 10, 2017

There is a thread about this on the forum which has more discussion in it.

@ncw ncw added the question label Aug 10, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment