DELAY attribute specifically for retries

[abmyii]

I assumed the DELAY attr would set the delay for retries but instead it applies to all requests. I would appreciate it if there was a DELAY attr specifically for retries (RETRY_DELAY). I'd be happy to implement it if given the go-ahead.

Thank you for this great library!

[howie6879]

Hi @abmyii ：
Did you mean setting a specific delay configuration during the retry process?

[abmyii]

Yes - but one that applies only for retries - not for all requests.

[howie6879]

I don’t quite understand, it possible to set retry every time when you use request? abmyii <notifications@github.com>于2020年1月1日 周三下午5:14写道：

…

Yes - but one that applies only for retries - not for all requests. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#80?email_source=notifications&email_token=AECB6XC3IYCQAY7EBIU5NJLQ3RNIPA5CNFSM4KBYLRX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH5BCLA#issuecomment-570036524>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AECB6XHXCJSG6B3GZAM2BCDQ3RNIPANCNFSM4KBYLRXQ> .

[abmyii]

How would I go about doing that?

[howie6879]

https://github.com/howie6879/ruia/blob/master/examples/simple_spider/douban_spider.py We assume a parameter named request_config in this spider demo. This setting can take effect in every request, Is this what you need? abmyii <notifications@github.com>于2020年1月1日 周三下午5:39写道：

…

How would I go about doing that? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#80?email_source=notifications&email_token=AECB6XC3WWKKQSX5D7EALQ3Q3RQERA5CNFSM4KBYLRX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH5BNNY#issuecomment-570037943>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AECB6XHLG7R2EHRL6GWWJXTQ3RQERANCNFSM4KBYLRXQ> .

[abmyii]

Here is a rough explanation:

request 1 (fails) -> wait for <x> seconds then retry -> request 2 (fails) -> repeat until request passes

But only for that request - the rest of the concurrent requests should continue like normal - with no timeout.

[howie6879]

For any request, the first step is to delay first. If it fails, then try again, and concurrent requests will also be delayed first. abmyii <notifications@github.com>于2020年1月1日 周三下午5:51写道：

…

Here is a rough explanation: request 1 (fails) -> wait for <x> seconds then retry -> request 2 (fails) -> repeat until request passes But *only* for that request - the rest of the concurrent requests should continue like normal - with no timeout. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#80?email_source=notifications&email_token=AECB6XCUCBEN7I6CZM2ETVLQ3RRSBA5CNFSM4KBYLRX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH5BSWY#issuecomment-570038619>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AECB6XEPH5SO3DFOOMNEMFTQ3RRSBANCNFSM4KBYLRXQ> .

[abmyii]

Yes. Can this be changed so that there is a delay only for retries (gives the server a chance to serve the page requested - i.e. if not cached already) so that the requests that get an immediate response aren't slowed down?

I might make a quick implementation to show what I mean. Will that help?

[howie6879]

Of course, it would be nice if there is code combined with the description, thank you abmyii <notifications@github.com>于2020年1月1日 周三下午6:17写道：

…

Yes. Can this be changed so that there is a delay *only* for retries (gives the server a chance to serve the page requested - i.e. if not cached already) so that the requests that get an immediate response aren't slowed down? I might make a quick implementation to show what I mean. Will that help? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#80?email_source=notifications&email_token=AECB6XABJANZCZWF3IAZ5L3Q3RUR3A5CNFSM4KBYLRX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH5B6MI#issuecomment-570040113>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AECB6XERAESY2BU24Q4TIOTQ3RUR3ANCNFSM4KBYLRXQ> .

[abmyii]

Done:
master...abmyii:retry_delay

[howie6879]

You want to set a special parameter for all retry requests to sleep If you think this demand is very strong, I think it can be added, can you submit a pull request？ abmyii <notifications@github.com>于2020年1月1日 周三下午6:23写道：

…

Done: master...abmyii:retry_delay <master...abmyii:retry_delay> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#80?email_source=notifications&email_token=AECB6XDUCOXKGTXJJC745LDQ3RVIHA5CNFSM4KBYLRX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH5CBMQ#issuecomment-570040498>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AECB6XDX4LICSHPIHH446GTQ3RVIHANCNFSM4KBYLRXQ> .

[abmyii]

Sure. I think the delay should be inside if self.retry_times > 0:, correct?
Also, thank you very much for taking the time to discuss this!

[howie6879]

Of course, retry only takes effect if a retry occurs，we both gave our time to this issue, and I want to thank you, too, for trying to make Ruia better abmyii <notifications@github.com>于2020年1月1日 周三下午6:32写道：

…

Sure. I think the delay should be inside if self.retry_times > 0:, correct? Also, thank you very much for taking the time to discuss this! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#80?email_source=notifications&email_token=AECB6XCG3A6NW7SXX6KGHNTQ3RWMHA5CNFSM4KBYLRX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH5CGLI#issuecomment-570041133>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AECB6XESRDAHNCLUA5WVUJ3Q3RWMHANCNFSM4KBYLRXQ> .