Add StatsContributorsStream #71
Conversation
| state_partitioning_keys = ["repo", "org"] | ||
| # Note - these queries are expensive and the API might return an HTTP 202 if the response | ||
| # has not been cached recently. https://docs.github.com/en/rest/reference/metrics#a-word-about-caching | ||
| tolerated_http_errors = [202] |
There was a problem hiding this comment.
@edgarrmondragon the docs say that the request should be retried "a bit later" when we get a 202. Is there some sort of mechanism in the sdk to do this (I'd imagine something like enqueueing retries after N seconds)? I guess it could be handle the same way as a 429 - Too many requests but that would less than ideal.
tap_github/repository_streams.py
Outdated
| weekly_data = contributor_activity["weeks"] | ||
| for week in weekly_data: | ||
| # no need to save weeks with no contributions. | ||
| is_week_empty = sum(week[key] for key in ["a", "c", "d"]) > 0 |
There was a problem hiding this comment.
How do we distinguish a missing response (say if we received a 202 that has not been retried yet, something like undefined) from this situation where it's a zero, in the downstream processing?
There was a problem hiding this comment.
I don't think that's relevant, we get data for the past year each time, so ultimately we will get the data we want. The only issue is if we ALWAYS get a 202...
tap_github/repository_streams.py
Outdated
| th.Property("org", th.StringType), | ||
| th.Property("user_id", th.IntegerType), | ||
| # Activity keys | ||
| th.Property("w", th.IntegerType), |
There was a problem hiding this comment.
I'd be tempted to rename these fields so they're more or less self-documenting:
week_startadditionsdeletionscommits
What do you think?
tap_github/repository_streams.py
Outdated
| th.Property("c", th.IntegerType), | ||
| # Contributor keys | ||
| th.Property("login", th.StringType), | ||
| th.Property("id", th.IntegerType), |
There was a problem hiding this comment.
Is id always the same as user_id here?
There was a problem hiding this comment.
I renamed id to user_id in the spirit of clarity as above.
tap_github/repository_streams.py
Outdated
| # Parent keys | ||
| th.Property("repo", th.StringType), | ||
| th.Property("org", th.StringType), | ||
| th.Property("user_id", th.IntegerType), |
There was a problem hiding this comment.
Do we actually receive user_id from the parent, if the parent is a repo?
Co-authored-by: Laurent Savaete <laurent@where.tf>
|
Kudos, SonarCloud Quality Gate passed!
|
Add the repo
/stats/contributorsendpoint to fetch weekly contribution data