Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Google Photos sync getting 429 and doesn't seem to backoff. #4935

Closed
philips opened this issue Jan 19, 2021 · 3 comments
Closed

Google Photos sync getting 429 and doesn't seem to backoff. #4935

philips opened this issue Jan 19, 2021 · 3 comments
Labels
question Remote: Google Photos waiting for reply...

Comments

@philips
Copy link
Contributor

philips commented Jan 19, 2021

What is the problem you are having with rclone?

Google Photos sync getting 429 and doesn't seem to backoff.

xref #4396

What is your rclone version (output from rclone version)

$ rclone version
rclone v1.53.3

  • os/arch: darwin/amd64
  • go version: go1.15.5

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

macos 11.1

Which cloud storage system are you using? (e.g. Google Drive)

google photos

The command you were trying to run (e.g. rclone copy /tmp remote:tmp)

rclone copy photos:media/by-month /home/foo/photos

A log from the command with the -vv flag (e.g. output from rclone -vv copy /tmp remote:tmp)

First 429

2021/01/16 20:09:27 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/01/16 20:09:27 DEBUG : HTTP REQUEST (req 0xc002c45700)
2021/01/16 20:09:27 DEBUG : GET REDACTED HTTP/1.1
Host: lh3.googleusercontent.com
User-Agent: rclone/v1.53.3
Authorization: XXXX
Accept-Encoding: gzip

2021/01/16 20:09:27 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/01/16 20:09:27 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/01/16 20:09:27 DEBUG : HTTP RESPONSE (req 0xc001ed8300)
2021/01/16 20:09:27 DEBUG : HTTP/2.0 429 Too Many Requests
Alt-Svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Sun, 17 Jan 2021 04:09:27 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

2021/01/16 20:09:27 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/01/16 20:09:27 DEBUG : pacer: low level retry 1/1 (error Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute per user' of service 'photoslibrary.googleapis.com' for consumer 'project_number:656339434255'. (429 RESOURCE_EXHAUSTED))
2021/01/16 20:09:27 DEBUG : pacer: Rate limited, increasing sleep to 1.789719605s
2021/01/16 20:09:27 DEBUG : Google Photos path "media/by-month": >List: err=couldn't list files: Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute per user' of service 'photoslibrary.googleapis.com' for consumer 'project_number:656339434255'. (429 RESOURCE_EXHAUSTED)
2021/01/16 20:09:27 ERROR : 2018/2018-11: error reading source directory: couldn't list files: Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute per user' of service 'photoslibrary.googleapis.com' for consumer 'project_number:656339434255'. (429 RESOURCE_EXHAUSTED)
2021/01/16 20:09:27 DEBUG : Google Photos path "media/by-month": List: dir="2003/2003-11"
2021/01/16 20:09:27 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

First 429 with a 403 following

2021/01/16 21:06:35 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/01/16 21:06:35 DEBUG : HTTP REQUEST (req 0xc001118000)
2021/01/16 21:06:35 DEBUG : GET /lr/REDACTED

HTTP/1.1
Host: lh3.googleusercontent.com
User-Agent: rclone/v1.53.3
Authorization: XXXX
Accept-Encoding: gzip

2021/01/16 21:06:35 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/01/16 21:06:35 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/01/16 21:06:35 DEBUG : HTTP RESPONSE (req 0xc001118000)
2021/01/16 21:06:35 DEBUG : HTTP/2.0 429 Too Many Requests
Content-Length: 1557
Access-Control-Allow-Origin: *
Alt-Svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma
=2592000,quic=":443"; ma=2592000; v="46,43"
Content-Type: text/html; charset=UTF-8
Date: Sun, 17 Jan 2021 05:06:35 GMT
Server: fife
Timing-Allow-Origin: *
Vary: Origin
X-Content-Type-Options: nosniff
X-Xss-Protection: 0

2021/01/16 21:06:35 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/01/16 21:06:35 DEBUG : pacer: low level retry 1/1 (error <!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 429 (Too Many Requests)!!1</title>
etc, etc, etc
  <p>The rate limit for this service has been exceeded.  <ins>That’s all we know.</ins>
 (429 429 Too Many Requests)
2021/01/16 21:06:35 DEBUG : 2006/2006-12/cimg0299%20(Modified%20(2)).jpg: Size: 
2021/01/16 21:06:35 DEBUG : 2006/2006-12/cimg0299%20(Modified%20(2)).jpg: >Size: 
2021/01/16 21:06:35 DEBUG : 2006/2006-12/cimg0299%20(Modified%20(2)).jpg: Size: 
2021/01/16 21:06:35 DEBUG : 2006/2006-12/cimg0299%20(Modified%20(2)).jpg: >Size: 
2021/01/16 21:06:35 DEBUG : 2019/2019-06/IMG_20190620_183439.jpg: Size: 
2021/01/16 21:06:35 DEBUG : 2019/2019-06/IMG_20190620_183439.jpg: >Size: 
2021/01/16 21:06:35 DEBUG : 2006/2006-12/cimg0299%20(Modified%20(2)).jpg: Size: 
2021/01/16 21:06:35 DEBUG : 2006/2006-12/cimg0299%20(Modified%20(2)).jpg: >Size: 
2021/01/16 21:06:35 DEBUG : 2006/2006-12/cimg0299%20(Modified%20(2)).jpg: Open: 
2021/01/16 21:06:51 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/01/16 21:06:51 DEBUG : HTTP REQUEST (req 0xc001118500)
2021/01/16 21:06:51 DEBUG : GET /lr/REDACTED HTTP/1.1
Host: lh3.googleusercontent.com
User-Agent: rclone/v1.53.3
Authorization: XXXX
Accept-Encoding: gzip

2021/01/16 21:06:51 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/01/16 21:06:51 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/01/16 21:06:51 DEBUG : HTTP RESPONSE (req 0xc001118500)
2021/01/16 21:06:51 DEBUG : HTTP/2.0 403 Forbidden
Alt-Svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Date: Sun, 17 Jan 2021 05:06:51 GMT
Server: fife
X-Content-Type-Options: nosniff
X-Xss-Protection: 0

2021/01/16 21:06:51 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/01/16 21:06:51 DEBUG : pacer: Reducing sleep to 0s
2021/01/16 21:06:51 DEBUG : 2006/2006-12/cimg0298-1.jpg: >Open: 
2021/01/16 21:06:51 ERROR : 2006/2006-12/cimg0298-1.jpg: Failed to copy: failed to open source object: <!DOCTYPE html><html lang=en><meta charset=utf-8><meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width"><title>Error 403 (Forbidden)!!1</title><style>*
Forbidden

Timestamp summary of the situation.

2021/01/16 21:06:35 DEBUG : HTTP/2.0 429 Too Many Requests
2021/01/16 21:06:51 DEBUG : HTTP/2.0 403 Forbidden
2021/01/16 21:07:08 DEBUG : HTTP/2.0 429 Too Many Requests
2021/01/16 21:07:08 DEBUG : HTTP/2.0 403 Forbidden
2021/01/16 21:07:08 DEBUG : HTTP/2.0 403 Forbidden
2021/01/16 21:07:08 DEBUG : HTTP/2.0 403 Forbidden
2021/01/16 21:07:08 DEBUG : HTTP/2.0 403 Forbidden
2021/01/16 21:07:09 DEBUG : HTTP/2.0 429 Too Many Requests
2021/01/16 21:07:09 DEBUG : HTTP/2.0 403 Forbidden
2021/01/16 21:07:09 DEBUG : HTTP/2.0 403 Forbidden
2021/01/16 21:07:09 DEBUG : HTTP/2.0 403 Forbidden
2021/01/16 21:07:09 DEBUG : HTTP/2.0 403 Forbidden
2021/01/16 21:07:11 DEBUG : HTTP/2.0 429 Too Many Requests
2021/01/16 21:07:11 DEBUG : HTTP/2.0 403 Forbidden
2021/01/16 21:07:11 DEBUG : HTTP/2.0 429 Too Many Requests
2021/01/16 21:07:11 DEBUG : HTTP/2.0 429 Too Many Requests
2021/01/16 21:07:11 DEBUG : HTTP/2.0 403 Forbidden
2021/01/16 21:07:13 DEBUG : HTTP/2.0 403 Forbidden
2021/01/16 21:07:13 DEBUG : HTTP/2.0 429 Too Many Requests
2021/01/16 21:07:13 DEBUG : HTTP/2.0 429 Too Many Requests
2021/01/16 21:07:13 DEBUG : HTTP/2.0 429 Too Many Requests
2021/01/16 21:07:13 DEBUG : HTTP/2.0 429 Too Many Requests
2021/01/16 21:07:13 DEBUG : HTTP/2.0 429 Too Many Requests
2021/01/16 21:07:15 DEBUG : HTTP/2.0 429 Too Many Requests
2021/01/16 21:07:32 DEBUG : HTTP/2.0 429 Too Many Requests
2021/01/16 21:07:48 DEBUG : HTTP/2.0 403 Forbidden
2021/01/16 21:08:04 DEBUG : HTTP/2.0 403 Forbidden
2021/01/16 21:08:04 DEBUG : HTTP/2.0 429 Too Many Requests
2021/01/16 21:08:04 DEBUG : HTTP/2.0 403 Forbidden
2021/01/16 21:08:04 DEBUG : HTTP/2.0 403 Forbidden
2021/01/16 21:08:04 DEBUG : HTTP/2.0 429 Too Many Requests
2021/01/16 21:08:04 DEBUG : HTTP/2.0 403 Forbidden
2021/01/16 21:08:06 DEBUG : HTTP/2.0 429 Too Many Requests
2021/01/16 21:08:06 DEBUG : HTTP/2.0 429 Too Many Requests
2021/01/16 21:08:06 DEBUG : HTTP/2.0 429 Too Many Requests
2021/01/16 21:08:06 DEBUG : HTTP/2.0 429 Too Many Requests
2021/01/16 21:08:06 DEBUG : HTTP/2.0 403 Forbidden
2021/01/16 21:08:08 DEBUG : HTTP/2.0 403 Forbidden
2021/01/16 21:08:08 DEBUG : HTTP/2.0 403 Forbidden
2021/01/16 21:08:08 DEBUG : HTTP/2.0 429 Too Many Requests
2021/01/16 21:08:08 DEBUG : HTTP/2.0 403 Forbidden
2021/01/16 21:08:08 DEBUG : HTTP/2.0 429 Too Many Requests
2021/01/16 21:08:08 DEBUG : HTTP/2.0 429 Too Many Requests
2021/01/16 21:08:08 DEBUG : HTTP/2.0 429 Too Many Requests
2021/01/16 21:08:08 DEBUG : HTTP/2.0 429 Too Many Requests
2021/01/16 21:08:10 DEBUG : HTTP/2.0 429 Too Many Requests
@philips
Copy link
Contributor Author

philips commented Jan 19, 2021

I have a full 940417 line debug log too. I am still trying to figure it all out but it seems like rclone isn't really respecting the 429s.

My debug command line: rclone copy photos:media/by-month /Users/brandonphilips/GooglePhotos/ --dump headers -vv --low-level-retries 1 --retries 1

@ncw
Copy link
Member

ncw commented Jan 22, 2021

Rclone respects the 429 errors but it is doing lots of things in parallel so the errors come in clumps.

You can try --checkers 1 --transfers 1 to slow rclone down.

You can also try --tpslimit 10 (experiment with the value) - that will help too.

@ivandeex
Copy link
Member

ivandeex commented Feb 8, 2021

Closing as the question was answered

@ivandeex ivandeex closed this as completed Feb 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Remote: Google Photos waiting for reply...
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@philips @ncw @ivandeex