HTTPError: 429 Client Error: Too many requests

[ptrstn]

In particular, when running the unit tests, deepl.com sometimes returns an Error 429: Too many requests.

Structured testing should be done to determine when this error occurs and what time periods are allowed without returning an error.

Another approach to fix this is to add a retry_after argument to wait a certain number of milliseconds between each request.

[ptrstn]

Another solution could be to implement support for proxies as suggested in: ytdl-org/youtube-dl#22839

[ptrstn]

A potential problem could be that the generated data always contains the exact same id and not the generated one.

deepl-translate/deepl/generators.py

Lines 5 to 14 in ca61c63

	def generate_split_sentences_request_data(text, identifier=MAGIC_NUMBER, **kwargs):
	return {
	"jsonrpc": "2.0",
	"method": "LMT_split_into_sentences",
	"params": {
	"texts": [text],
	"lang": {"lang_user_selected": "auto", "user_preferred_langs": []},
	},
	"id": identifier,
	}

deepl-translate/deepl/generators.py

Lines 41 to 64 in ca61c63

	def generate_translation_request_data(
	source_language,
	target_language,
	sentences,
	identifier=MAGIC_NUMBER,
	alternatives=1,
	formality_tone=None,
	):
	return {
	"jsonrpc": "2.0",
	"method": "LMT_handle_jobs",
	"params": {
	"jobs": generate_jobs(sentences, beams=alternatives),
	"lang": {
	"user_preferred_langs": [target_language, source_language],
	"source_lang_computed": source_language,
	"target_lang": target_language,
	},
	"priority": 1,
	"commonJobParams": generate_common_job_params(formality_tone),
	"timestamp": generate_timestamp(sentences),
	},
	"id": identifier,
	}

deepl-translate/deepl/hacks.py

Lines 21 to 22 in ca61c63

	def generate_id():
	return randrange(1_000_000, 100_000_000)

This id should probably be maintained and incremented in a translation session.

[KevinZonda]

A potential problem could be that the generated data always contains the exact same id and not the generated one.

deepl-translate/deepl/generators.py

Lines 5 to 14 in ca61c63

	def generate_split_sentences_request_data(text, identifier=MAGIC_NUMBER, **kwargs):
	return {
	"jsonrpc": "2.0",
	"method": "LMT_split_into_sentences",
	"params": {
	"texts": [text],
	"lang": {"lang_user_selected": "auto", "user_preferred_langs": []},
	},
	"id": identifier,
	}

deepl-translate/deepl/generators.py

Lines 41 to 64 in ca61c63

	def generate_translation_request_data(
	source_language,
	target_language,
	sentences,
	identifier=MAGIC_NUMBER,
	alternatives=1,
	formality_tone=None,
	):
	return {
	"jsonrpc": "2.0",
	"method": "LMT_handle_jobs",
	"params": {
	"jobs": generate_jobs(sentences, beams=alternatives),
	"lang": {
	"user_preferred_langs": [target_language, source_language],
	"source_lang_computed": source_language,
	"target_lang": target_language,
	},
	"priority": 1,
	"commonJobParams": generate_common_job_params(formality_tone),
	"timestamp": generate_timestamp(sentences),
	},
	"id": identifier,
	}

deepl-translate/deepl/hacks.py

Lines 21 to 22 in ca61c63

	def generate_id():
	return randrange(1_000_000, 100_000_000)

This id should probably be maintained and incremented in a translation session.

Hi, I did some research and believe it is a generated field. Could try decompile the Windows client to get some inspiration

[ptrstn]

Hi, I did some research and believe it is a generated field. Could try decompile the Windows client to get some inspiration

Yes, it is generated and the javascript is quite simply.

utils.chunk.js:1511

var l = 1e4 * Math.round(1e4 * Math.random())

utils.chunk.js:1532

return function h(y, m) {
    var g = arguments.length > 2 && void 0 !== arguments[2] ? arguments[2] : 0
        , b = ++l
        , w = {
        jsonrpc: "2.0",
        method: t,
        params: void 0 !== y ? y : {},
        id: b
    }

In Python:

def generate_id() -> int:
    """
    From utils.chunk.$16023f.js:formatted:1511
    var l = 1e4 * Math.round(1e4 * Math.random())

    :return:
    """
    return int(1e4) * round(1e4 * random.random())

I still experienced the HTTP 429 error though. Not sure when I will find some time to do some experiments.

[KevinZonda]

also deepl uses fake timestamp

Try below

fn gen_fake_timestamp(texts: &Vec<String>) -> u128 {
    let ts = tool::get_epoch_ms();
    let i_count = texts
            .iter()
            .fold(
                1, 
                |s, t| s + t.text.matches('i').count()
            ) as u128;
    ts - ts % i_count + i_count
}

if u know chinese, try read https://zu1k.com/posts/thinking/deception-tactics-in-deepl-api-design/

[ptrstn]

also deepl uses fake timestamp

Try below

fn gen_fake_timestamp(texts: &Vec<String>) -> u128 {
    let ts = tool::get_epoch_ms();
    let i_count = texts
            .iter()
            .fold(
                1, 
                |s, t| s + t.text.matches('i').count()
            ) as u128;
    ts - ts % i_count + i_count
}

if u know chinese, try read https://zu1k.com/posts/thinking/deception-tactics-in-deepl-api-design/

Yes, thanks. Its already implemented in https://github.com/ptrstn/deepl-translate/blob/master/deepl/hacks.py

[JounQin]

@ptrstn

See https://github.com/rx-ts/deepl-translate/blob/master/src/api.ts#L30-L40

Stringified "method":" need to be changed a bit. 😂