Retry strategy and use semaphores to prevent creating a large number of async tasks

[onkarbhardwaj]

---

Status

READY

Description

• As a workaround for temporary network outages and temporary deployment issues, we are introducing retry with exponential backoff for asynchronous versions of generate and tokenize.
• We are also introducing semphores to prevent a large number of simultaneous async tasks getting created in case a user thinks of sending hundreds of thousands of prompts.
• Semaphores also enable Retry strategy in a subtle way by preventing pushing a request-to-be-retried behind a large number of async queries. If not prevented, such a query could potentially be pushed behind all these tasks and that would affect run-time for ordered mode of generate_async.
• Semaphores also enabled running much larger number of prompts in async manner.

Impacted Areas in Library

AsyncResponseGenerator, RequestHandler.

Which issue(s) does this pull-request fix?

Closes: #15

Any special notes for your reviewer?

For testing the effect of semaphores, try sending a large number of prompts (>200k) with or without these changes. The exact number might differ system-to-system. For testing retry-strategy, check the dev example in this PR. This example randomly alters http return code and causes retry+backoff.

---

Checklist

• Automated tests exist
• Updated Package Requirements (if required, and with maintainers' approval)
• Local unit tests performed
• Documentation exists link
• Local pre-commit hooks performed
• Desired commit message set as PR title and description set above
• Link to relevant GitHub issue provided

[coveralls]

Pull Request Test Coverage Report for Build 5417755269

• 32 of 32 (100.0%) changed or added relevant lines in 2 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.08%) to 94.737%

---

Totals
Change from base Build 5379768299:	0.08%
Covered Lines:	1242
Relevant Lines:	1311

---

💛 - Coveralls

[mirianfsilva]

lgtm!

[onkarbhardwaj]

Resolved merge conflict via GUI. Hope tabs don't create an indentation issue.

Has merge conflicts

[mirianfsilva]

tests are not passing after merge with develop. But the async examples works fine with +550 prompts.

[mirianfsilva]

LGTM, and works fine with +1000 prompts.