Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Try to use GitHub PAT when fetching forecast dates; and check response status for rate limit #649

Merged
merged 2 commits into from
Jul 10, 2023

Conversation

nmdefries
Copy link
Contributor

@nmdefries nmdefries commented Jul 10, 2023

Followup to #647. Because dates are now being fetched using the GitHub API, scripts that request a lot of dates (e.g. forecast eval pipeline) can run into rate limits. When the API is rate-limited, get_covidhub_forecast_dates raises a non-informative warning and returns empty dates, causing downstream failures that are hard to debug.

Look for a GITHUB_PAT environment variable first, then GITHUB_TOKEN if *_PAT doesn't exist. If either is found, use it to authenticate the GET requests (using this header format). Response status codes are checked and if a 403 status is returned, the function exits with an error. It doesn't make sense to warn and return anyway because the results are empty; or to retry with backoff because the rate limit resets after an hour.

Note: I used the API's message as the error message because I'm not sure a 403-status is unique to reaching the rate limit and I want the actual error to be clear, and checking the text of the message wouldn't be robust.

Copy link
Collaborator

@dshemetov dshemetov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great, thanks for taking care of this!

i checked locally that both auth and unauth requests work and that the error comes through well 👍

:shipit: :shipit: :shipit:

@dshemetov dshemetov merged commit 1d505e3 into main Jul 10, 2023
4 checks passed
@dshemetov dshemetov deleted the ndefries/use-gh-auth-forecast-dates branch July 10, 2023 22:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants