New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle pagination of results #255

Closed
walquis opened this Issue Jun 7, 2013 · 8 comments

Comments

Projects
None yet
6 participants
@walquis

walquis commented Jun 7, 2013

It would be great if the client could abstract away the pagination headaches, a la https://github.com/jwilger/github-v3-api. The current version only returns the first 100 repos,

The only way to get all the results is to parse the "Link" header for the "since" arg, and pass it to the next request, etc., stringing results together until there are no more "Link" headers with "since" args.

@fs-build-system

This comment has been minimized.

Show comment
Hide comment
@fs-build-system

fs-build-system Jun 11, 2013

Agreed. Ran into that problem today. By default auto traversal (aka: auto paginate) is disabled. You can change the default by setting the 'auto_traversal' option to true when authenticating. In ruby:

client = Octokit::Client.new(oauth_token: , auto_traversal: true)

fs-build-system commented Jun 11, 2013

Agreed. Ran into that problem today. By default auto traversal (aka: auto paginate) is disabled. You can change the default by setting the 'auto_traversal' option to true when authenticating. In ruby:

client = Octokit::Client.new(oauth_token: , auto_traversal: true)

@sporkmonger

This comment has been minimized.

Show comment
Hide comment
@sporkmonger

sporkmonger Aug 2, 2013

👍

A more elegant solution is really needed, particularly given that there's a rate limit on the API and auto_traversal will hit that rate limit effortlessly. Which incidentally, also means that you need the ability to resume after some kind of exponential back-off.

sporkmonger commented Aug 2, 2013

👍

A more elegant solution is really needed, particularly given that there's a rate limit on the API and auto_traversal will hit that rate limit effortlessly. Which incidentally, also means that you need the ability to resume after some kind of exponential back-off.

@pengwynn

This comment has been minimized.

Show comment
Hide comment
@pengwynn

pengwynn Aug 2, 2013

Member

In addition to the auto traversal approach, Octokit 2 (now in release candidate) provides access to responses, including headers, and even parses Link relations:

gem install octokit --pre
>> repos = Octokit.repos 'sporkmonger'
>> rels = Octokit.last_response.rels
{
    :next => #<Sawyer::Relation: next: get https://api.github.com/user/1778/repos?page=2>,
    :last => #<Sawyer::Relation: last: get https://api.github.com/user/1778/repos?page=2>
}
>> rels[:next].get.data

Let us know if you fancy something different. Patches welcome.

Member

pengwynn commented Aug 2, 2013

In addition to the auto traversal approach, Octokit 2 (now in release candidate) provides access to responses, including headers, and even parses Link relations:

gem install octokit --pre
>> repos = Octokit.repos 'sporkmonger'
>> rels = Octokit.last_response.rels
{
    :next => #<Sawyer::Relation: next: get https://api.github.com/user/1778/repos?page=2>,
    :last => #<Sawyer::Relation: last: get https://api.github.com/user/1778/repos?page=2>
}
>> rels[:next].get.data

Let us know if you fancy something different. Patches welcome.

@pengwynn pengwynn closed this Aug 2, 2013

@sporkmonger

This comment has been minimized.

Show comment
Hide comment
@sporkmonger

sporkmonger Aug 2, 2013

I also just noticed this:

client = Octokit::Client.new(oauth_token: token, auto_traversal: true, per_page: 100)
client.rate_limit - client.rate_limit_remaining
# => 48
commits = client.commits("octokit/octokit.rb")
client.rate_limit - client.rate_limit_remaining
# => 59 # 11 requests used
client = Octokit::Client.new(oauth_token: token, auto_traversal: true, per_page: 500)
commits = client.commits("octokit/octokit.rb")
client.rate_limit - client.rate_limit_remaining
# => 70 # 11 requests used

It appears that changing the number of items returned per page does not result in fewer total requests made. Not sure why.

sporkmonger commented Aug 2, 2013

I also just noticed this:

client = Octokit::Client.new(oauth_token: token, auto_traversal: true, per_page: 100)
client.rate_limit - client.rate_limit_remaining
# => 48
commits = client.commits("octokit/octokit.rb")
client.rate_limit - client.rate_limit_remaining
# => 59 # 11 requests used
client = Octokit::Client.new(oauth_token: token, auto_traversal: true, per_page: 500)
commits = client.commits("octokit/octokit.rb")
client.rate_limit - client.rate_limit_remaining
# => 70 # 11 requests used

It appears that changing the number of items returned per page does not result in fewer total requests made. Not sure why.

@pengwynn

This comment has been minimized.

Show comment
Hide comment
@pengwynn

pengwynn Aug 2, 2013

Member

There is an API limit of 100 items per page. Setting auto_traversal to true will set the page size to 100 for maximum efficiency.

Member

pengwynn commented Aug 2, 2013

There is an API limit of 100 items per page. Setting auto_traversal to true will set the page size to 100 for maximum efficiency.

@sporkmonger

This comment has been minimized.

Show comment
Hide comment
@sporkmonger

sporkmonger Aug 2, 2013

Ah hah, that's why it did the same thing for 35. OK.

sporkmonger commented Aug 2, 2013

Ah hah, that's why it did the same thing for 35. OK.

@Dorian

This comment has been minimized.

Show comment
Hide comment
@Dorian

Dorian Jan 6, 2017

Doesn't seem to work with on the organization's members API for instance, e.g.:

>> Octokit.org('google').rels[:public_members].get.data.size
=> 30
>> Octokit.auto_paginate = true
=> true
>> Octokit.org('google').rels[:public_members].get.data.size
=> 30

Dorian commented Jan 6, 2017

Doesn't seem to work with on the organization's members API for instance, e.g.:

>> Octokit.org('google').rels[:public_members].get.data.size
=> 30
>> Octokit.auto_paginate = true
=> true
>> Octokit.org('google').rels[:public_members].get.data.size
=> 30
@AnriKoul

This comment has been minimized.

Show comment
Hide comment
@AnriKoul

AnriKoul Jan 6, 2017

Octokit.auto_paginate = true

AnriKoul commented Jan 6, 2017

Octokit.auto_paginate = true

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment