Recover translation fail due to "HTTP Error 429: Too Many Requests"

[killo3967]

Hi.

Some times using Google or DeepL free, i get the error "HTTP Error 429: Too Many Requests".
I know....I know it's my fault to want to translate a lot.

But, It might be possible when this happens that in the next translation process of this book I ask for a new start or do I continue the previous translation? In this way it could be possible to translate a book several times.

Also, I think it could also be interesting that when this error's happens, it could be possible to set a time interval (for example hours) to automatically continue the translation.

The log is:

Original: Mike was born on 5 March 1942. He sold his first article in 1957, his first short story in 1959 and his first book in 1962.
Traceback (most recent call last):
File "calibre_plugins.ebook_translator.engines.base", line 89, in get_result.
File "mechanize_mechanize.py", line 257, in open
File "mechanize_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class..httperror_seek_wrapper: HTTP Error 429: Too Many Requests
The translation engine API data could not be retrieved.
Will try again in 5 seconds.
Retrying... (timeout is 300 seconds).
Traceback (most recent call last):
File "calibre_plugins.ebook_translator.engines.base", line 89, in get_result.
File "mechanize_mechanize.py", line 257, in open
File "mechanize_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class..httperror_seek_wrapper: HTTP Error 429: Too Many Requests

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "calibre_plugins.ebook_translator.translation", line 65, in _translate_text
File "calibre_plugins.ebook_translator.engines.google", line 25, in translate
File "calibre_plugins.ebook_translator.engines.base", line 100, in get_result
Exception: Could not parse the returned response. Raw Data: HTTP Error 429: Too Many Requests

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "calibre_plugins.ebook_translator.engines.base", line 89, in get_result
File "mechanize_mechanize.py", line 257, in open
File "mechanize_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class..httperror_seek_wrapper: HTTP Error 429: Too Many Requests
Could not retrieve data from the translation engine API.
Will try again in 10 seconds.
Retrying... (timeout is 300 seconds).
Traceback (most recent call last):
File "calibre_plugins.ebook_translator.engines.base", line 89, in get_result.
File "mechanize_mechanize.py", line 257, in open
File "mechanize_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class..httperror_seek_wrapper: HTTP Error 429: Too Many Requests

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "calibre_plugins.ebook_translator.translation", line 65, in _translate_text
File "calibre_plugins.ebook_translator.engines.google", line 25, in translate
File "calibre_plugins.ebook_translator.engines.base", line 100, in get_result
Exception: Could not parse the returned response. Raw Data: HTTP Error 429: Too Many Requests

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "calibre_plugins.ebook_translator.engines.base", line 89, in get_result
File "mechanize_mechanize.py", line 257, in open
File "mechanize_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class..httperror_seek_wrapper: HTTP Error 429: Too Many Requests

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "calibre_plugins.ebook_translator.translation", line 65, in _translate_text
File "calibre_plugins.ebook_translator.engines.google", line 25, in translate
File "calibre_plugins.ebook_translator.engines.base", line 100, in get_result
Exception: Could not parse the returned response. Raw Data: HTTP Error 429: Too Many Requests

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "calibre_plugins.ebook_translator.engines.base", line 89, in get_result
File "mechanize_mechanize.py", line 257, in open
File "mechanize_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class..httperror_seek_wrapper: HTTP Error 429: Too Many Requests
The translation engine API data could not be retrieved.
Will try again in 30 seconds.

[bookfere]

Don't worry if you turn on the cache feature in settings, because it will recover translated content from the local cache whenever translation processing is interrupted for any reason.

We already have an "Attempt times" setting that allows you to wait for a long time and retry failed attempts. The retry count and attempt interval have a calculated relationship and follow this pattern:

1 x 5 = 5 seconds
2 x 5 = 10 seconds
3 x 10 = 30 seconds
4 x 30 = 120 seconds (2 minutes)
5 x 120 = 600 seconds (10 minutes)
6 x 600 = 3600 seconds (1 hour)
7 x 3600 = 25200 seconds (7 hours)
...

So, if you set the "Attempt times" to 6, it will try to make the request six times with an increasing interval, and when it reaches the sixth attempt, it will wait for 1 hour before retrying.

I'm unsure if this will meet your needs, so please don't hesitate to let me know if it doesn't.

[joelucky198x]

同样的问题，翻译了一半出现这个，貌似是太多的请求，被服务器被拒绝了，哈哈。

[killo3967]

Could you put a timestamp in the log file to check the times of retry. beacuse i tried to translate the same book with the parameter "Attempt times" in 6 and have the same problem. Now i am translating with the parameter in 7 secons.

[wangshuihua]

OK, no problem. We will add a timestamp in the next version.

Could you put a timestamp in the log file to check the times of retry. beacuse i tried to translate the same book with the parameter "Attempt times" in 6 and have the same problem. Now i am translating with the parameter in 7 secons.

[bookfere]

Many users on MobileRead have also encountered the 'Too Many Requests' issue. After some investigation, we discovered that the problem may have been caused by our decision to switch the request method of Google (Free) from GET to POST in version 1.3.0 and later. This change was intended to support larger character counts, but it may have inadvertently triggered the issue. As a result, we plan to revert back to the GET method in future versions.

[bookfere]

Version 1.3.3 has been released. Please give it a try.

[killo3967]

Thanks very much for your investigation. I cheched the other day 1.2.2 version and could trnslate a book in 15 minutes instead of with the 1.3.1 version which don't finish because this problem. There is another problem about the finger print of the user that the deepl and google could obtain from the browsers and block the users. Is there any probability that you could implement any code about this?

[bookfere]

Could you give me some information on the errors related to the "fingerprint" problem?

[killo3967]

Forgive me if I have explained myself badly. What I am really referring to is an issue of anonymity. When you make an http request to a website this site is able to take information from you and create what is called a fingerprint, this way you get the famous message "Too many request". There are ways to anonymise the communication, but I understand that this would not be within the scope of your project. If you are interested in more information you can visit "https://coveryourtracks.eff.org". I am currently working on a proxy+tor system to prevent deepl or google from tracking you.

Best regards

[bookfere]

Thank you for providing more details about anonymity. If this idea could be implemented, it could potentially offer a better solution to the "Too many requests" issue. Keep up the good work 👍