Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exception in tumblr extractor after waiting for rate limit #4916

Closed
Sketch6307 opened this issue Dec 12, 2023 · 1 comment
Closed

Exception in tumblr extractor after waiting for rate limit #4916

Sketch6307 opened this issue Dec 12, 2023 · 1 comment
Labels

Comments

@Sketch6307
Copy link

$ gallery-dl --verbose https://www.tumblr.com/thevaultofass
[gallery-dl][debug] Version 1.26.4-dev
[gallery-dl][debug] Python 3.11.5 - nope
[gallery-dl][debug] requests 2.31.0 - urllib3 2.1.0
[gallery-dl][debug] Configuration Files ['/etc/gallery-dl.conf']
[gallery-dl][debug] Starting DownloadJob for 'https://www.tumblr.com/thevaultofass'
[tumblr][debug] Using TumblrUserExtractor for 'https://www.tumblr.com/thevaultofass'
[tumblr][debug] Using custom OAuth1.0 authentication
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): api.tumblr.com:443
[urllib3.connectionpool][debug] https://api.tumblr.com:443 "GET /v2/blog/thevaultofass.tumblr.com/posts?limit=50&reblog_info=true HTTP/1.1" 429 150
[tumblr][debug] {'meta': {'status': 429, 'msg': 'Limit Exceeded'}, 'response': [], 'errors': [{'title': 'Limit Exceeded', 'code': 0, 'detail': 'Measly little error. Try again.'}]}
[tumblr][info] Hourly API rate limit exceeded
[tumblr][info] Waiting until 06:10:01 for rate limit reset.
[tumblr][error] An unexpected error occurred: TypeError - TumblrAPI._call() got multiple values for argument 'params'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[tumblr][debug] 
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/gallery_dl/job.py", line 127, in run
    for msg in extractor:
  File "/usr/lib/python3.11/site-packages/gallery_dl/extractor/tumblr.py", line 81, in items
    for post in self.posts():
  File "/usr/lib/python3.11/site-packages/gallery_dl/extractor/tumblr.py", line 474, in _pagination
    data = self._call(endpoint, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/gallery_dl/extractor/tumblr.py", line 464, in _call
    return self._call(endpoint, params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: TumblrAPI._call() got multiple values for argument 'params'

This is perhaps not the neatest fix but it's what I did, since params is added to kwargs at the start of the function:

diff --git a/gallery_dl/extractor/tumblr.py b/gallery_dl/extractor/tumblr.py
index f50ddb79..648a331d 100644
--- a/gallery_dl/extractor/tumblr.py
+++ b/gallery_dl/extractor/tumblr.py
@@ -449,7 +449,7 @@ class TumblrAPI(oauth.OAuth1API):
 
                 if self.extractor.config("ratelimit") == "wait":
                     self.extractor.wait(seconds=reset)
-                    return self._call(endpoint, params, **kwargs)
+                    return self._call(endpoint, **kwargs)
 
                 t = (datetime.now() + timedelta(seconds=float(reset))).time()
                 raise exception.StopExtraction(
@@ -461,7 +461,7 @@ class TumblrAPI(oauth.OAuth1API):
             if reset:
                 self.log.info("Hourly API rate limit exceeded")
                 self.extractor.wait(seconds=reset)
-                return self._call(endpoint, params, **kwargs)
+                return self._call(endpoint, **kwargs)
 
         raise exception.StopExtraction(data)
@mikf mikf added the site:bug label Dec 12, 2023
mikf added a commit that referenced this issue Dec 12, 2023
use a loop instead of recursive function calls
@mikf
Copy link
Owner

mikf commented Dec 12, 2023

Your solution would have been OK, but this time I'd rather use a proper loop to retry API requests like it should have been done from the start.

@mikf mikf closed this as completed Dec 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants