List all Users + Gitlab 8.6 with python-gitlab #106

Asher256 · 2016-03-23T21:57:51Z

Hi Gauvain,

Before the update from Gitlab 8.5 to 8.6, python-gitlab was working perfectly.

But after the update to Gitlab 8.6, this code raised the exception below:

result = {}
for user in git.User(all=True):
    result[user.email.lower()] = user

The exception (It says it took too much time to respond, but the exception is raised instantaneously):

 File "./buggy.py", line 424, in <module>
    main()
  File "./buggy.py", line 420, in main
    add_missing_emails_to_projects(ldap_emails, gpg_emails, acl_emails)
  File "./buggy.py", line 218, in add_missing_emails_to_projects
    gitlab_users = gitlab_get_users()
  File "./buggy.py", line 201, in gitlab_get_users
    for user in git.User(all=True):
  File "/var/lib/work/.bin/python/gitlab/__init__.py", line 717, in User
    return User._get_list_or_object(self, id, **kwargs)
  File "/var/lib/work/.bin/python/gitlab/objects.py", line 279, in _get_list_or_object
    return cls.list(gl, **kwargs)
  File "/var/lib/work/.bin/python/gitlab/objects.py", line 246, in list
    return gl.list(cls, **kwargs)
  File "/var/lib/work/.bin/python/gitlab/__init__.py", line 398, in list
    results.extend(self.list(obj_class, **args))
  File "/var/lib/work/.bin/python/gitlab/__init__.py", line 398, in list
    results.extend(self.list(obj_class, **args))
  File "/var/lib/work/.bin/python/gitlab/__init__.py", line 398, in list
    results.extend(self.list(obj_class, **args))
  File "/var/lib/work/.bin/python/gitlab/__init__.py", line 398, in list
    results.extend(self.list(obj_class, **args))
  File "/var/lib/work/.bin/python/gitlab/__init__.py", line 376, in list
    raise_error_from_response(r, GitlabListError)
  File "/var/lib/work/.bin/python/gitlab/exceptions.py", line 135, in raise_error_from_response
    response_body=response.content)
gitlab.exceptions.GitlabListError: 502: b'<!DOCTYPE html>\n<html>\n<head>\n  <title>GitLab is not responding (502)</title>\n  <style>\n    body {\n      color: #666;\n      text-align: center;\n      font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;\n      margin: 0;\n      width: 800px;\n      margin: auto;\n      font-size: 14px;\n    }\n\n    h1 {\n      font-size: 56px;\n      line-height: 100px;\n      font-weight: normal;\n      color: #456;\n    }\n\n    h2 {\n      font-size: 24px;\n      color: #666;\n      line-height: 1.5em;\n    }\n\n    h3 {\n      color: #456;\n      font-size: 20px;\n      font-weight: normal;\n      line-height: 28px;\n    }\n\n    hr {\n      margin: 18px 0;\n      border: 0;\n      border-top: 1px solid #EEE;\n      border-bottom: 1px solid white;\n    }\n  </style>\n</head>\n<body>\n  <h1>\n    <img src="data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iMjEwIiBoZWlnaHQ9IjIxMCIgdmlld0JveD0iMCAwIDIxMCAyMTAiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+CiAgPHBhdGggZD0iTTEwNS4wNjE0IDIwMy42NTVsMzguNjQtMTE4LjkyMWgtNzcuMjhsMzguNjQgMTE4LjkyMXoiIGZpbGw9IiNlMjQzMjkiLz4KICA8cGF0aCBkPSJNMTA1LjA2MTQgMjAzLjY1NDhsLTM4LjY0LTExOC45MjFoLTU0LjE1M2w5Mi43OTMgMTE4LjkyMXoiIGZpbGw9IiNmYzZkMjYiLz4KICA8cGF0aCBkPSJNMTIuMjY4NSA4NC43MzQxbC0xMS43NDIgMzYuMTM5Yy0xLjA3MSAzLjI5Ni4xMDIgNi45MDcgMi45MDYgOC45NDRsMTAxLjYyOSA3My44MzgtOTIuNzkzLTExOC45MjF6IiBmaWxsPSIjZmNhMzI2Ii8+CiAgPHBhdGggZD0iTTEyLjI2ODUgODQuNzM0Mmg1NC4xNTNsLTIzLjI3My03MS42MjVjLTEuMTk3LTMuNjg2LTYuNDExLTMuNjg1LTcuNjA4IDBsLTIzLjI3MiA3MS42MjV6IiBmaWxsPSIjZTI0MzI5Ii8+CiAgPHBhdGggZD0iTTEwNS4wNjE0IDIwMy42NTQ4bDM4LjY0LTExOC45MjFoNTQuMTUzbC05Mi43OTMgMTE4LjkyMXoiIGZpbGw9IiNmYzZkMjYiLz4KICA8cGF0aCBkPSJNMTk3Ljg1NDQgODQuNzM0MWwxMS43NDIgMzYuMTM5YzEuMDcxIDMuMjk2LS4xMDIgNi45MDctMi45MDYgOC45NDRsLTEwMS42MjkgNzMuODM4IDkyLjc5My0xMTguOTIxeiIgZmlsbD0iI2ZjYTMyNiIvPgogIDxwYXRoIGQ9Ik0xOTcuODU0NCA4NC43MzQyaC01NC4xNTNsMjMuMjczLTcxLjYyNWMxLjE5Ny0zLjY4NiA2LjQxMS0zLjY4NSA3LjYwOCAwbDIzLjI3MiA3MS42MjV6IiBmaWxsPSIjZTI0MzI5Ii8+Cjwvc3ZnPgo=" /><br />\n    502\n  </h1>\n  <h3>Whoops, GitLab is taking too much time to respond.</h3>\n  <hr/>\n  <p>Try refreshing the page, or going back and attempting the action again.</p>\n  <p>Please contact your GitLab administrator if this problem persists.</p>\n</body>\n</html>\n'

Any idea Gauvain?

Thank you in advance!

Asher256

PS: I tried the same thing with pyapi-gitlab:

result = {}
for user in old_git.getall(old_git.getusers, per_page=100):
    result[user['email']] = user

And it listed the users without any error.

The text was updated successfully, but these errors were encountered:

Asher256 · 2016-03-23T23:14:19Z

I also have a warning:

/var/lib/work/.bin/python/gitlab/__init__.py:716: DeprecationWarning: `User` is deprecated, use `users` instead
  DeprecationWarning)

Asher256 · 2016-03-23T23:18:36Z

Another test:

In [17]: git.users.search(query='', all=True)
---------------------------------------------------------------------------
GitlabListError                           Traceback (most recent call last)
/var/lib/work/.hidden.git/scripts/gitlab_api/helper.py in <module>()
----> 1 git.users.search(query='', all=True)

/var/lib/work/.bin/python/gitlab/objects.py in search(self, query, **kwargs)
    570         """
    571         url = self.obj_cls._url + '?search=' + query
--> 572         return self.gitlab._raw_list(url, self.obj_cls, **kwargs)
    573 
    574     def get_by_username(self, username, **kwargs):

/var/lib/work/.bin/python/gitlab/__init__.py in _raw_list(self, path, cls, **kwargs)
    292             args = kwargs.copy()
    293             args['next_url'] = r.links['next']['url']
--> 294             results.extend(self.list(cls, **args))
    295         return results
    296 

/var/lib/work/.bin/python/gitlab/__init__.py in list(self, obj_class, **kwargs)
    396             args = kwargs.copy()
    397             args['next_url'] = r.links['next']['url']
--> 398             results.extend(self.list(obj_class, **args))
    399         return results
    400 

/var/lib/work/.bin/python/gitlab/__init__.py in list(self, obj_class, **kwargs)
    396             args = kwargs.copy()
    397             args['next_url'] = r.links['next']['url']
--> 398             results.extend(self.list(obj_class, **args))
    399         return results
    400 

/var/lib/work/.bin/python/gitlab/__init__.py in list(self, obj_class, **kwargs)
    396             args = kwargs.copy()
    397             args['next_url'] = r.links['next']['url']
--> 398             results.extend(self.list(obj_class, **args))
    399         return results
    400 

/var/lib/work/.bin/python/gitlab/__init__.py in list(self, obj_class, **kwargs)
    374                 "Can't connect to GitLab server (%s)" % e)
    375 
--> 376         raise_error_from_response(r, GitlabListError)
    377 
    378         cls = obj_class

/var/lib/work/.bin/python/gitlab/exceptions.py in raise_error_from_response(response, error, expected_code)
    133     raise error(error_message=message,
    134                 response_code=response.status_code,
--> 135                 response_body=response.content)

GitlabListError: 502: b'<!DOCTYPE html>\n<html>\n<head>\n  <title>GitLab is not responding (502)</title>\n  <style>\n    body {\n      color: #666;\n      text-align: center;\n      font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;\n      margin: 0;\n      width: 800px;\n      margin: auto;\n      font-size: 14px;\n    }\n\n    h1 {\n      font-size: 56px;\n      line-height: 100px;\n      font-weight: normal;\n      color: #456;\n    }\n\n    h2 {\n      font-size: 24px;\n      color: #666;\n      line-height: 1.5em;\n    }\n\n    h3 {\n      color: #456;\n      font-size: 20px;\n      font-weight: normal;\n      line-height: 28px;\n    }\n\n    hr {\n      margin: 18px 0;\n      border: 0;\n      border-top: 1px solid #EEE;\n      border-bottom: 1px solid white;\n    }\n  </style>\n</head>\n<body>\n  <h1>\n    <img src="data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iMjEwIiBoZWlnaHQ9IjIxMCIgdmlld0JveD0iMCAwIDIxMCAyMTAiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+CiAgPHBhdGggZD0iTTEwNS4wNjE0IDIwMy42NTVsMzguNjQtMTE4LjkyMWgtNzcuMjhsMzguNjQgMTE4LjkyMXoiIGZpbGw9IiNlMjQzMjkiLz4KICA8cGF0aCBkPSJNMTA1LjA2MTQgMjAzLjY1NDhsLTM4LjY0LTExOC45MjFoLTU0LjE1M2w5Mi43OTMgMTE4LjkyMXoiIGZpbGw9IiNmYzZkMjYiLz4KICA8cGF0aCBkPSJNMTIuMjY4NSA4NC43MzQxbC0xMS43NDIgMzYuMTM5Yy0xLjA3MSAzLjI5Ni4xMDIgNi45MDcgMi45MDYgOC45NDRsMTAxLjYyOSA3My44MzgtOTIuNzkzLTExOC45MjF6IiBmaWxsPSIjZmNhMzI2Ii8+CiAgPHBhdGggZD0iTTEyLjI2ODUgODQuNzM0Mmg1NC4xNTNsLTIzLjI3My03MS42MjVjLTEuMTk3LTMuNjg2LTYuNDExLTMuNjg1LTcuNjA4IDBsLTIzLjI3MiA3MS42MjV6IiBmaWxsPSIjZTI0MzI5Ii8+CiAgPHBhdGggZD0iTTEwNS4wNjE0IDIwMy42NTQ4bDM4LjY0LTExOC45MjFoNTQuMTUzbC05Mi43OTMgMTE4LjkyMXoiIGZpbGw9IiNmYzZkMjYiLz4KICA8cGF0aCBkPSJNMTk3Ljg1NDQgODQuNzM0MWwxMS43NDIgMzYuMTM5YzEuMDcxIDMuMjk2LS4xMDIgNi45MDctMi45MDYgOC45NDRsLTEwMS42MjkgNzMuODM4IDkyLjc5My0xMTguOTIxeiIgZmlsbD0iI2ZjYTMyNiIvPgogIDxwYXRoIGQ9Ik0xOTcuODU0NCA4NC43MzQyaC01NC4xNTNsMjMuMjczLTcxLjYyNWMxLjE5Ny0zLjY4NiA2LjQxMS0zLjY4NSA3LjYwOCAwbDIzLjI3MiA3MS42MjV6IiBmaWxsPSIjZTI0MzI5Ii8+Cjwvc3ZnPgo=" /><br />\n    502\n  </h1>\n  <h3>Whoops, GitLab is taking too much time to respond.</h3>\n  <hr/>\n  <p>Try refreshing the page, or going back and attempting the action again.</p>\n  <p>Please contact your GitLab administrator if this problem persists.</p>\n</body>\n</html>\n'

gpocentek · 2016-03-24T06:22:08Z

I don't really understand why this suddenly fails, and I can't test right now (hotel wifi...). I'll have a closer look as soon as I can.

Asher256 · 2016-03-25T16:17:45Z

Yes, it is a strange behaviour. Let me know ;-) . Thank you Gauvain.

FlorianKaiser · 2016-04-06T14:03:57Z

I encountered the same problem.

next_url somehow gets passed down through kwargs and is appended recursively to the url.
After a certain number of users (recursive paging), the request size gets too big which leads to the reported Gitlab error. Maybe the request size limit is more strict in Gitlab 8.6.

2016/04/06 11:36:02 [error] 12425#0: *11 upstream sent too big header while reading response header from upstream, client: 172.17.42.1, server: xxx, request: "GET /api/v3/users?all=True&next_url=http%3A%2F%2Flocalhost%3A1234%2Fapi%2Fv3%2Fusers%3Fall%3DTrue%26next_url%3Dhttp%253A%252F%252Flocalhost%253A1234%252Fapi%252Fv3%252Fusers%253Fall%253DTrue%2526 (...)

As a workaround, add the following in __init__.py:369 (in the list() function before the request ist sent):

if params.has_key('next_url'): del params['next_url']

Asher256 · 2016-04-06T15:01:26Z

+1

gdubicki · 2016-04-18T13:10:08Z

+1

I used code similar to this #93 (comment) as a workaround.

gpocentek · 2016-05-08T11:12:59Z

Thanks for the investigation. This seems to be fixed now.

gpocentek closed this as completed in 64af398 May 8, 2016

github-actions bot locked as resolved and limited conversation to collaborators Nov 29, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

List all Users + Gitlab 8.6 with python-gitlab #106

List all Users + Gitlab 8.6 with python-gitlab #106

Asher256 commented Mar 23, 2016

Asher256 commented Mar 23, 2016

Asher256 commented Mar 23, 2016

gpocentek commented Mar 24, 2016

Asher256 commented Mar 25, 2016

FlorianKaiser commented Apr 6, 2016

Asher256 commented Apr 6, 2016

gdubicki commented Apr 18, 2016

gpocentek commented May 8, 2016

List all Users + Gitlab 8.6 with python-gitlab #106

List all Users + Gitlab 8.6 with python-gitlab #106

Comments

Asher256 commented Mar 23, 2016

Asher256 commented Mar 23, 2016

Asher256 commented Mar 23, 2016

gpocentek commented Mar 24, 2016

Asher256 commented Mar 25, 2016

FlorianKaiser commented Apr 6, 2016

Asher256 commented Apr 6, 2016

gdubicki commented Apr 18, 2016

gpocentek commented May 8, 2016