RateLimitError #66

sarahjohns · 2016-09-21T02:53:12Z

Hi!

Thank you for the updates of the code. I tried to run the new updated version. After about 10 downloads, I receive the following traceback:

Traceback (most recent call last):
File "C:/Users/Documents/Python Scripts/collect_gtrends.py", line 34, in
trend=pytrend.trend(trend_payload, return_type='dataframe')
File "C:\Users\AppData\Roaming\Python\Python27\site-packages\pytrends\request.py", line 62, in trend
raise RateLimitError
pytrends.request.RateLimitError

I don't think this is the quota limit problem. Maybe I was downloading too frequently? How may seconds do you guys wait in between requests? My current program lets it sleep for 5-10 seconds. Is that not enough? Thank you!

sarahjohns · 2016-09-21T03:06:32Z

Actually, I tried again, now it gives the same traceback plus the following message:
Exceeded Google's Rate Limit. Please use time.sleep() to space requests.

I was using your old version of pytrends several months back. During that time, letting it sleep for 5-10 seconds worked well for me. Has Google increased the limit recently? If you know what the limit is, it would be very helpful for me. Thank you!

dreyco676 · 2016-09-21T13:07:52Z

I've been successful with 6 seconds. I think they allow 10/min

On Tue, Sep 20, 2016 at 10:06 PM, sarahjohns notifications@github.com
wrote:

Actually, I tried again, now it gives the same traceback plus the
following message:
Exceeded Google's Rate Limit. Please use time.sleep() to space requests.

I was using your old version of pytrends several months back. During that
time, letting it sleep for 5-10 seconds worked well for me. Has Google
increased the limit recently? If you know what the limit is, it would be
very helpful for me. Thank you!

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#66 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGxCB_psiY-oIl7bRW2tqY7IZ2qM7yOdks5qsJ84gaJpZM4KCWU4
.

sarahjohns · 2016-09-21T15:08:43Z

How many requests do you generally have? I remember that it used to be 1500 downloads per 24 hrs. Now it seems to be much lower. How many requests are you able to download with the 6 seconds wait in between? Thank you!

sarahjohns · 2016-09-21T16:16:38Z

I still experience the RateLimitError after 10 downloads even when I put time.sleep(30). One of the previous posts mentioned that they needed a cookie in the browser called PREF. Do you know how to add it? I could not figure out a way to add it... I would really appreciate it if you could help me figure out this issue.

dreyco676 · 2016-09-21T16:25:08Z

And you are for sure logged into your Gmail account? I'm able to chug along
at 10/min. I've put it a loop to see how long I can keep going.

On Wed, Sep 21, 2016 at 11:16 AM, sarahjohns notifications@github.com
wrote:

I still experience the RateLimitError after 10 downloads even when I put
time.sleep(30). One of the previous posts mentioned that they needed a
cookie in the browser called PREF. Do you know how to add it? I could not
figure out a way to add it... I would really appreciate it if you could
help me figure out this issue.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#66 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGxCB7PGAE1xkWHF9jcv-tpTS0cWf4mrks5qsVhngaJpZM4KCWU4
.

sarahjohns · 2016-09-21T16:57:32Z

I have a loop that goes down a list of inputs. Is it important that I call TrendReq(google_username, google_password, custom_useragent=None) inside the loop, i.e., before every request?

dreyco676 · 2016-09-21T18:15:30Z

No I don't think you would need to relogin each time. I just tested as was
able to run 203 requests spaced at 10/min but then it finally gave me the
error. So if I had to guess its probably 10/min, 200/hr.

# connect to Google
pytrend = TrendReq(google_username, google_password, custom_useragent='My Pytrends Script')

trend_payload = {'q': 'Pizza, Italian, Spaghetti, Breadsticks, Sausage', 'cat': '0-71'}

i = 1
while i <= 3000:
    # trend
    trend = pytrend.trend(trend_payload)
    #print(trend)
    print(i)
    time.sleep(6)
    i += 1

sarahjohns · 2016-09-21T18:52:35Z

I see. Thank you a lot!

senielee · 2016-09-25T15:16:20Z

I used time.sleep(randint(20, 30)) and I got Rate Limit error after 10 request. That seems weird or google change the rate limit again?

urataj · 2016-09-27T15:44:46Z

I've found that I'm limited to 12 requests per hour.
However, for some time this morning I've received the following response:
Response did not parse. See server response for details. Sorry, our systems are a little stressed out right now and need to take a deep breath. Please try again in a few moments.

cdshinde · 2016-09-28T03:42:12Z

Yes, I too saw this response, when I was trying to run the "https://www.google.com/trends/fetchComponent" URL with its required query parameters from the browser. Though it was working normally from the Google Trends URL, for the same URL/IP combination.

paulwohlfarth · 2016-10-12T12:36:39Z

I'm receiving the same error, funnily enough when scraping the data only. I could still download reports manually without issues even after I hit a suspected rate limit. I also agree with urataj, the rate limit, if that's what's causing this, must be much lower than 200/hr. Could there be some 24hr limit as well? I've also noticed that some of the data seems to be compromised. By that I don't mean the occasional sampling error but large parts of the data having zero values or being otherwise completely spurious.

dreyco676 · 2016-10-19T15:19:30Z

I'm guessing that the library has a bug that is not correctly passing the session/cookies to google and hits an extremely low rate limit. I will look into it when I've got time. Until then if this is a showstopper check out a version before v3.0 it has a clunkier method to get the data.

paulwohlfarth · 2016-10-19T22:22:14Z

For all I know it is a showstopper for now as the URL used to scrape has changed completely. There is now a token added to the end, which looks very much like the session id's used in the developer platform for the official google api's (at least that's the way it works for maps apparently). And I believe this token hasn't been there before. I wonder if this means that we might get an official trends api soon though? I believe it was planned originally but pretty much forgotten for years. Let's hope the guys at google caught up with that now.

gmlee7 · 2016-12-07T05:17:41Z

I have been struggling for two days on the rate limit. I should have read this thread before starting. I also found that the rate limit is very low: After 12 calls (with 30-second intervals), I got a rate limit error. I wish Google supports official API soon...

colincolinkn · 2016-12-07T23:22:32Z

Getting this error again... Can you help us to solve it soon? Thanks!
Response did not parse. See server response for details.

patrickongwong · 2016-12-29T03:18:18Z

Getting similar error... but it worked for the first maybe 4 queries.
Thanks for the great work though!

Michelangelo1994 · 2017-01-24T22:33:44Z

Hi, I have a similar problem here. Can anyone help us out to resolve this issue? Thanks.

dreyco676 · 2017-01-24T23:29:34Z

I spent some time investigating today. It looks like they changed the API and perhaps throttled the old one.

https://www.google.com/trends/api/widgetdata/multiline?

It looks like it has a ton more parameters that I'll have to wade through but hopefully using that I can resolve this.

dreyco676 · 2017-01-24T23:43:55Z

Here is the URL for getting CSV's directly for Pizza & Bagel

https://www.google.com/trends/api/widgetdata/multiline/csv?req=%7B%22time%22%3A%222012-01-24%202017-01-24%22%2C%22resolution%22%3A%22WEEK%22%2C%22locale%22%3A%22en-US%22%2C%22comparisonItem%22%3A%5B%7B%22geo%22%3A%7B%7D%2C%22complexKeywordsRestriction%22%3A%7B%22keyword%22%3A%5B%7B%22type%22%3A%22BROAD%22%2C%22value%22%3A%22pizza%22%7D%5D%7D%7D%2C%7B%22geo%22%3A%7B%7D%2C%22complexKeywordsRestriction%22%3A%7B%22keyword%22%3A%5B%7B%22type%22%3A%22BROAD%22%2C%22value%22%3A%22bagel%22%7D%5D%7D%7D%5D%2C%22requestOptions%22%3A%7B%22property%22%3A%22%22%2C%22backend%22%3A%22IZG%22%2C%22category%22%3A0%7D%7D&token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&tz=360

dreyco676 · 2017-01-24T23:45:27Z

URL decodes to:

https://www.google.com/trends/api/widgetdata/multiline/csv?req=
{
   "time":"2012-01-24 2017-01-24",
   "resolution":"WEEK",
   "locale":"en-US",
   "comparisonItem":[
      {
         "geo":{

         },
         "complexKeywordsRestriction":{
            "keyword":[
               {
                  "type":"BROAD",
                  "value":"pizza"
               }
            ]
         }
      },
      {
         "geo":{

         },
         "complexKeywordsRestriction":{
            "keyword":[
               {
                  "type":"BROAD",
                  "value":"bagel"
               }
            ]
         }
      }
   ],
   "requestOptions":{
      "property":"",
      "backend":"IZG",
      "category":0
   }
}
&token=XXXXXXXXXXXXXXX
&tz=360

dreyco676 · 2017-01-25T00:06:30Z

Looks like you can maybe get the the Token from this url request

https://www.google.com/trends/api/explore?hl=en-US&tz=360&req=%7B%22comparisonItem%22:%5B%7B%22keyword%22:%22pizza%22,%22geo%22:%22%22,%22time%22:%22today+5-y%22%7D,%7B%22keyword%22:%22bagel%22,%22geo%22:%22%22,%22time%22:%22today+5-y%22%7D%5D,%22category%22:0,%22property%22:%22%22%7D&tz=360

which decodes to

https://www.google.com/trends/api/explore?hl=en-US&tz=360&req=
{
   "comparisonItem":[
      {
         "keyword":"pizza",
         "geo":"",
         "time":"today 5-y"
      },
      {
         "keyword":"bagel",
         "geo":"",
         "time":"today 5-y"
      }
   ],
   "category":0,
   "property":""
}
&tz=360

Cool we might have something to work with....

dreyco676 · 2017-01-25T05:04:02Z

I'm able to pull back data this way but it is not a trivial fix and will take some time.

cwallac · 2017-01-25T14:25:25Z

@dreyco676 absolutely fantastic, thanks for discovering that.

dreyco676 · 2017-01-25T14:40:45Z

In case people are curious as to how I'm doing this. I'm using Fiddler to inspect the web requests that go across when using my browser. From there its all about making PyTrends mimic those requests.

jblemoine · 2017-01-25T14:55:44Z

@dreyco676 Awesome !!! Thanks a lot.

jblemoine · 2017-01-25T15:01:39Z

@dreyco676 How did you get the url where I can find the token ? I would like to select a specific country.

dreyco676 · 2017-01-25T15:08:23Z

Fiddler Just look for a URL that starts with https://www.google.com/trends/api/explore?hl=en-US&tz=360

You can specify the country with the geo param.

dreyco676 · 2017-01-25T17:00:28Z

Alright I was able to test that I have request limit that is in the 100's. I will close this issue as it will be fixed in the next release.

This was referenced Jan 25, 2017

You have reached your quota limit #68

Closed

Google Quota limit - IP Address changer #77

Closed

dreyco676 closed this as completed Jan 25, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RateLimitError #66

RateLimitError #66

sarahjohns commented Sep 21, 2016 •

edited

sarahjohns commented Sep 21, 2016

dreyco676 commented Sep 21, 2016

sarahjohns commented Sep 21, 2016

sarahjohns commented Sep 21, 2016

dreyco676 commented Sep 21, 2016

sarahjohns commented Sep 21, 2016

dreyco676 commented Sep 21, 2016 •

edited

sarahjohns commented Sep 21, 2016

senielee commented Sep 25, 2016

urataj commented Sep 27, 2016

cdshinde commented Sep 28, 2016

paulwohlfarth commented Oct 12, 2016

dreyco676 commented Oct 19, 2016

paulwohlfarth commented Oct 19, 2016

gmlee7 commented Dec 7, 2016

colincolinkn commented Dec 7, 2016

patrickongwong commented Dec 29, 2016 •

edited

Michelangelo1994 commented Jan 24, 2017

dreyco676 commented Jan 24, 2017

dreyco676 commented Jan 24, 2017

dreyco676 commented Jan 24, 2017 •

edited

dreyco676 commented Jan 25, 2017 •

edited

dreyco676 commented Jan 25, 2017

cwallac commented Jan 25, 2017

dreyco676 commented Jan 25, 2017

jblemoine commented Jan 25, 2017

jblemoine commented Jan 25, 2017

dreyco676 commented Jan 25, 2017

dreyco676 commented Jan 25, 2017

RateLimitError #66

RateLimitError #66

Comments

sarahjohns commented Sep 21, 2016 • edited

sarahjohns commented Sep 21, 2016

dreyco676 commented Sep 21, 2016

sarahjohns commented Sep 21, 2016

sarahjohns commented Sep 21, 2016

dreyco676 commented Sep 21, 2016

sarahjohns commented Sep 21, 2016

dreyco676 commented Sep 21, 2016 • edited

sarahjohns commented Sep 21, 2016

senielee commented Sep 25, 2016

urataj commented Sep 27, 2016

cdshinde commented Sep 28, 2016

paulwohlfarth commented Oct 12, 2016

dreyco676 commented Oct 19, 2016

paulwohlfarth commented Oct 19, 2016

gmlee7 commented Dec 7, 2016

colincolinkn commented Dec 7, 2016

patrickongwong commented Dec 29, 2016 • edited

Michelangelo1994 commented Jan 24, 2017

dreyco676 commented Jan 24, 2017

dreyco676 commented Jan 24, 2017

dreyco676 commented Jan 24, 2017 • edited

dreyco676 commented Jan 25, 2017 • edited

dreyco676 commented Jan 25, 2017

cwallac commented Jan 25, 2017

dreyco676 commented Jan 25, 2017

jblemoine commented Jan 25, 2017

jblemoine commented Jan 25, 2017

dreyco676 commented Jan 25, 2017

dreyco676 commented Jan 25, 2017

sarahjohns commented Sep 21, 2016 •

edited

dreyco676 commented Sep 21, 2016 •

edited

patrickongwong commented Dec 29, 2016 •

edited

dreyco676 commented Jan 24, 2017 •

edited

dreyco676 commented Jan 25, 2017 •

edited