Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Project listing agains taiga.io fails: too many projects #59

Closed
erikw opened this issue Oct 28, 2017 · 3 comments
Closed

Project listing agains taiga.io fails: too many projects #59

erikw opened this issue Oct 28, 2017 · 3 comments
Labels
type: feature

Comments

@erikw
Copy link
Contributor

erikw commented Oct 28, 2017

Hello,

I noticed that if you use python-taiga against https://api.taiga.io/ to list projects it fails.

api.projects.list()
(Traceback (most recent call last):
  File "./taiga-stats", line 175, in cmd_list_projects
    for proj in api.projects.list():
  File "/home/erikw/.virtualenvs/taiga-stats/lib/python3.6/site-packages/taiga/models/base.py", line 41, in list
    objects = self.parse_list(result.json())
  File "/home/erikw/.virtualenvs/taiga-stats/lib/python3.6/site-packages/requests/models.py", line 892, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

No output fed in to the JSON decoding? So I decided to get down to my friend command line, and this works as expected:

$ curl -X GET -H "Content-Type: application/json" -H "Authorization: Bearer ${AUTH_TOKEN}" -s https://api.taiga.io/api/v1/projects
...
<project JSON>

So I drilled down the code and noticed that python-taiga always send the header

x-disable-pagination: True

to disable pagination and get everything at once.

So I plugged this in to the the previous command line

$ curl -X GET -H "Content-Type: application/json" -H "Authorization: Bearer ${AUTH_TOKEN}" -H "x-disable-pagination: True" -s https://api.taiga.io/api/v1/projects
<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.10.3 (Ubuntu)</center>
</body>
</html>

and now it it also fails like when called with pyhon-taiga.

I simply think this is the situation: now the number of public projects at taiga.io has recently grown too large to be included in one GET request to the project listing API endpoint. But before it was small enough to work.

A user of my project taiga-stats reported this last week.

I guess python-taiga should be adapted to use pagination for requests, so it can handle large taiga instances, like taiga.io now is. And I think this is prioritized to to, as I guess taiga.io is the most used instance of Taiga of them all!
@yakky
Copy link
Member

yakky commented Oct 28, 2017

Nice investigation!
We are happy that taiga.io now exceeds this limit :)
We definitely need to implement pagination support, I'm wondering what the most useful way to do it would be: any suggestion?

@erikw
Copy link
Contributor Author

erikw commented Oct 29, 2017

Yes me too, happy that there are so many projects :).

I was thinking shortly about this. I guess we have a few options

  • Don't expose functions to the users that requires pagination, like api.projects.list()
  • Keep the current interface for python-taiga and hide the fact that things are paginated and collected in the background before returning.
  • Make a new major version of pyhon-taiga which exposes the pagination to the user, so the user of the library have the burden or the possibility to deal with pagination.

I think option 1 is out of the picture.

But I think a combination of options 2 and 3 would be good; keep the current library functions, but make them fetch all pages if the results returned from the first HTTP GET exceeds one page. AND introduce new functions which allows the user to fetch one page per time. This way, I think we would cover the most use cases for this library!

@yakky yakky mentioned this issue Dec 30, 2017
Merged
@yakky
Copy link
Member

yakky commented Dec 30, 2017

@erikw what do you think of the preliminary pagination API. It implements your suggestion regarding a mix of 2 and 3 in a backward compatible way
Tests are needed, but I am interested in your feedback on the API details

@yakky yakky closed this as completed in #61 Jan 10, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type: feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@erikw @yakky