Quota limit and -S option

[luc4sdreyer]

I was polling Google trends at about one request every 70 seconds, which is pretty slow in my opinion. Then I got the following error:

org.freaknet.gtrends.api.exceptions.GoogleTrendsClientException: 
*** You are running too fast man! Looks like you reached your quota limit. 
Wait a while and slow it down with the '-S' option! ***

at
org.freaknet.gtrends.api.GoogleTrendsClient.execute(GoogleTrendsClient.java:72)

I couldn't find any information about the -S option in the source code. Is it still supported? And does anyone know what a safe speed is?

[elibus]

Which version are you using? Can you provide the full cmdline?

[luc4sdreyer]

I'm using j-google-trends-client-1.2.6-jar-with-dependencies.jar

I'm not sure I understand about the cmdline. I'm using the example in the readme, except without a proxy. It works fine, but after 3 calls I get the error message.

[elibus]

Please note the -S option take ms and not seconds

[ikaros1223]

Dear developer,
I have downloaded your the newest file and have executed including -S option. However I've got the same exception errors like Luc4sdreyer.
*** You are running too fast man! Looks like you reached your quota limit. Wait a while and slow it down with the '-S' option! ***
Is it possible to be blocked by Google? Even though I tried several times with different sleep time options such as 50000, 60000 and 70000 I've got the same errors. Would you please give me any opinions about that? thank you so much!

[elibus]

google has changed the service making it not working anymore. there is a new version on its way but still not production ready.
Anyway looks like they restricted the service use even more.

[ikaros1223]

Thank you for your fast reply. If you are going to find some co-workers who can develop another version you've mentioned then could I get a chance to join developing the new version if it possible?
Have a good day !

[elibus]

Knock yourself out! I have already commited everything, if you like to contribute just fork and pull

[kamir]

Hi Marco,

do you have any idea where I can find more details about the existing
limitations? I need some numbers to explain the limitations in the Google
Trends API. What is the maximum number of calls per time frame and what is
the maximum volume and so on.

The tool I work on needs keyword trends for about 10.000 terms in one
analysis step. Using Wikipedia I have a working solution now the goal is to
extend this to a more generic data source or even to explain why we can not
go with this tool to Google Trends data.

Is the an official commercial service available for Google Trends data?
Could not find much on that on the web so far.
Many thanks.

Best wishes,
Mirko

On Mon, Aug 4, 2014 at 9:24 AM, Marco Tizzoni notifications@github.com
wrote:

google has changed the service making it not working anymore. there is a
new version on its way but still not production ready.
Anyway looks like they restricted the service use even more.

—
Reply to this email directly or view it on GitHub
#7 (comment)
.

Mirko Kämpf

Trainer @ Cloudera

tel: +49 176 20 63 51 99
skype: kamir1604
mirko@cloudera.com

[elibus]

No clue on limitations, sorry. Google is not disclosing such informat. AFAIK restrictions have been tighten over last year.

There is no commercial version og GT I am aware of, you should contect Google.

[kamir]

Thanks a lot. So I am more convinced not to have missed something.
Best wishes
Mirko

On Mon, Aug 4, 2014 at 1:06 PM, Marco Tizzoni notifications@github.com
wrote:

No clue on limitations, sorry. Google is not disclosing such informat.
AFAIK restrictions have been tighten over last year.

There is no commercial version og GT I am aware of, you should contect
Google.

—
Reply to this email directly or view it on GitHub
#7 (comment)
.

Mirko Kämpf

Trainer @ Cloudera

tel: +49 176 20 63 51 99
skype: kamir1604
mirko@cloudera.com

[luc4sdreyer]

In my experience it is easier to parse and pull data from the website than use the API, but even that is severely limited.

It is actually very unfortunate that Google doesn't allow others to leverage the power of the Trends API. I've wondered why but I can't come up with an answer. It is highly likely that some of the smart guys at Google have tried to fix the API, and I'm sure there are many companies out there that would pay huge sums of money for a proper API. This is not some old business like an ancient bank we're talking about here, it is one of the most innovative companies in the world.

There are a couple of other services out there that claim to offer something similar (just google "Google Trends API alternative"), but I doubt that anyone really could offer a comparing product.

I ended up using Twitter's API instead, it is absolutely brilliant and almost unlimited.

[kamir]

Many thanks Lucas. I see your point, and I am looking for some facts to us
in a publication which explains our approach using Wikipedia data - so far
I was feeling, I write just about a personal conclusion or opinion. But
some people do "data science" on Google Trends data. They do not write how
they get the data and I think this is big limitation already. They even do
not publish their data sources and data sets, so it is not easy to trace
back results.
But anyway, here we can not solve the problem, but maybe we can find a
comparable solution, using Twitter or Wikipedia or even both together.
Cheers,
Mirko

On Mon, Aug 4, 2014 at 1:27 PM, Lucas Dreyer notifications@github.com
wrote:

In my experience it is easier to parse and pull data from the website than
use the API, but even that is severely limited.

It is actually very unfortunate that Google doesn't allow others to
leverage the power of the Trends API. I've wondered why but I can't come up
with an answer. It is highly likely that some of the smart guys at Google
have tried to fix the API, and I'm sure there are many companies out there
that would pay huge sums of money for a proper API. This is not some old
business like an ancient bank we're talking about here, it is one of the
most innovative companies in the world.

There are a couple of other services out there that claim to offer
something similar (just google "Google Trends API alternative"), but I
doubt that anyone really could offer a comparing product.

I ended up using Twitter's API instead, it is absolutely brilliant and
almost unlimited.

—
Reply to this email directly or view it on GitHub
#7 (comment)
.

Mirko Kämpf

Trainer @ Cloudera

tel: +49 176 20 63 51 99
skype: kamir1604
mirko@cloudera.com

[elibus]

Yhey do it manually. That's why I wrote this tool. Some colleague of mine wrote a paper and they did everything manually. I thought it was inefficient.

The issue with GT is that NO proper API is released from Google. What my API does is simply emulating a browser and exposing an interface where people can automate the task of downloading data. Unfortunately as there is no API there is no certainty on compatibility, in fact I have already adopted my API to minor changes Google made over the time, and they try to block bots.

The last resort is doing full browser emulation, still the problem is that they have reduced the amount of query a user can do daily hence dunno if that is worth a shot.

[kamir]

Marco, I think the last statement is really important, as soon as we need
1.000.000 time series for a field study, we
are limited. The manual approach seems to be fine in the beginning, but
later as soon as you start with normalization of data you need much more
for good statistical results. But we are far away from this.

On Mon, Aug 4, 2014 at 1:39 PM, Marco Tizzoni notifications@github.com
wrote:

Yhey do it manually. That's why I wrote this tool. Some colleague of mine
wrote a paper and they did everything manually. I thought it was
inefficient.

The issue with GT is that NO proper API is released from Google. What my
API does is simply emulating a browser and exposing an interface where
people can automate the task of downloading data. Unfortunately as there is
no API there is no certainty on compatibility, in fact I have already
adopted my API to minor changes Google made over the time, and they try to
block bots.

The last resort is doing full browser emulation, still the problem is that
they have reduced the amount of query a user can do daily hence dunno if
that is worth a shot.

—
Reply to this email directly or view it on GitHub
#7 (comment)
.

Mirko Kämpf

Trainer @ Cloudera

tel: +49 176 20 63 51 99
skype: kamir1604
mirko@cloudera.com