Enhanced API calls: set User-Agent header & E-Utility rate limits #83
Conversation
On my system, this produced: manubot/0.2.1 (Linux; Python/3.6) <contact@manubot.org>
|
Interesting that the AppVeyor build failed the test: Status code 429 refers to too many requests. |
Thanks. When there are potential alternate reviewers I appreciate having them help out. I'll still watch and casually review all pull requests.
Is this an issue? Do we know how many requests are allowed? |
Adopt sytem specific files from manubot/rootstock@135f451 With the addition of .vscode
|
Looks like the issue could be that E-utilities API has a limit of 3 requests per second. More info here. If you register for an API key, that increases to 10 requests per second. I wonder why this was never an issue in the past, but perhaps some aspect of the tests on AppVeyor got faster, which is causing us to hit the limit. I'll look into a workaround. |
|
This is a naive idea, but could it be related to AppVeyor running tests in parallel in multiple environments? |
This commit is now a separate PR This reverts commit 0653c8e.
dhimmel
changed the title
| @@ -241,7 +245,7 @@ def get_pmcid_and_pmid_for_doi(doi): | |||
| return {} | |||
| try: | |||
| element_tree = xml.etree.ElementTree.fromstring(response.text) | |||
| except Exception as error: | |||
| except Exception: | |||
There was a problem hiding this comment.
Don't you want to include the exception as a string in the logger?
There was a problem hiding this comment.
In this case, I don't think the exception will be helpful, because it will be a very verbose and unwieldy stacktrace that the XML could not be parsed. We log response.text, which will usually contain a non-XML error message that is actually relevant.
There was a problem hiding this comment.
Sounds good. I am going to approve this PR.
|
@agitter does this PR look okay? I don't think you've seen it since I added the rate limiter for E-Utility queries. |
CC @RazerM... thanks! The context manager is helpful since we can apply a single ratelimiter across calls in multiple functions. |
Adds new manubot contact email to user-agent, which can be helpful so resources can contact us regarding our API requests. Adds platform and python version info to user-agent so the upcoming translation-server can monitor usage.