Skip to content

Add support for GitHub requests which involve pagination #1107

@begedin

Description

@begedin

Problem

Our api needs support for pagination in order to deal with initial syncing of github requests

http://justincalleja.com/2016/01/26/elixir-streams-reload/#How_to_fetch_Github_organization_repos_and_their_pagination has a pretty nice writeup on how we could potentially implement that.

However, while streams can be done in paralel, the write up in that link does it sequentially, since it relies on links.next_page to fetch each page of results.

GitHub does not recommended this, but we definitely could

  • get the first page to figure out how many pages there are

  • create a parallel request for each page

  • entrypoint should be CodeCorps.GitHub

  • since pagination is on GET requests, there should not be a need to support specifying the method as an argument - get_all or something like that would work

  • it's used for syncing, so it should be an integration request

Once we have that, we could continue by adding higher level functions to our individual endpoints, in separate issues.

  • GitHub.API.Installation.repositories may have to be enhanced to support this
  • GitHub.API.Repository.issues needs to be implemented
  • GitHub.API.Issue.comments needs to be implemented

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions