PaginatedList.totalCount KeyError: 'page' with large results

[Twitch]

When using get_repo().totalCount on a small result set (only my own repositories) totalCount returns the repository count correctly. When querying my enterprise Github for a complete list of repositories, the following error occurs:

Traceback (most recent call last):
  File "./test-repo-list.py", line 9, in <module>
    print("Found %s repositories." % full_repo_list.totalCount)
  File "/usr/local/lib/python3.7/dist-packages/github/PaginatedList.py", line 175, in totalCount
    self.__totalCount = int(parse_qs(lastUrl)['page'][0])
KeyError: 'page'

I am using Python 3.7.1 and the pip-installed version 1.43.4 of PyGithub with a Github Enterprise version 2.14.7.

This is the code I have used to test this:

#!/usr/bin/env python3.7
from github import Github
import os

api_token = os.getenv('GITHUB_TOKEN', None)
g = Github(base_url="https://mygithubhost.lads", login_or_token=api_token)

full_repo_list = g.get_repos()
print("Found %s repositories." % full_repo_list.totalCount)

I did notice, after some skimming of PaginatedList.py, that the Link headers differ between the two tests. When collecting just my own repositories, I see the "last" link returned in the link header, but when querying for all repositories I only see "next" and "first" links returned.

g.get_user().get_repos():

'link':'<https://mygithubhost.lads/api/v3/user/repos?per_page=1&page=2>; rel="next", <https://mygithubhost.lads/api/v3/user/repos?per_page=1& 
page=10>; rel="last"'`

g.get_repos():

'link':'<https://mygithubhost.lads/api/v3/repositories?per_page=1&since=112>; rel="next", <https://mygithubhost.lads/api/v3/repositories{?since}>; rel="first"'

I don't know what's causing this difference in behavior, though.

If I don't try to get totalCount from the object, it works as expected, but totalCount specifically seems to fail.

[Hanaasagi]

Maybe, From the list-all-public-repositories API doc.

Note: Pagination is powered exclusively by the since parameter. Use the Link header to get the URL for the next page of repositories.

It not support per_page parameter, so the pagination strategy is not working. But I don't know why there is no last` link returned in the link header.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stoiev]

This issue could be related to the way that PaginatedList.py parses links urls from headers.

parse_qs is executed using a full url. It fails parsing first parameter (rigth after ?)

>>> from urllib.parse import parse_qs
>>> parse_qs("https://github.com/Cobliteam/?page=11")
{'https://github.com/Cobliteam/?page': ['11']}
>>> parse_qs("https://github.com/Cobliteam/?per_page=3&page=11")
{'https://github.com/Cobliteam/?per_page': ['3'], 'page': ['11']}

Could this bug be reopened? Or should I open a new issue?

[SlapDrone]

Hey, just got stung by this. Anyone since managed to find a solution?

[EnricoMi]

Never seen this, reopening this issue for tracking.

[sag1sh-rezilion]

I've had a few tackles with this issue, it reappears more when using the lazy=True when creating the repo object (as an example - r = g.get_repo('docker/docker', lazy=True); c = r.get_contributors(anon='true'); c.totalCount

An easy fix could be in the function totalCount in the line self.__totalCount = int(parse_qs(lastUrl)["page"][0])
instead there should be -
urlQuery = urlparse(lastUrl).query
self.__totalCount = int(parse_qs(urlQuery)["page"][0])