Support gitlabs link headers to fetch all results

[thomasf]

The gitlab api has a http header named Link ( http://www.w3.org/wiki/LinkHeader ) which looks like this for a first result page:

<http://gitlab.../api/v3/issues?page=2&per_page=0>; rel="next", <http://gitlab.../api/v3/issues?page=1&per_page=0>; rel="first", <http://gitlab.../api/v3/issues?page=12&per_page=0>; rel="last"

If alot of requests might be required to reach a full result list emacs-gitlab should probably be converted to async/lexical scope first so that emacs doesn't freeze up while reqests are being fetched.

[thomasf]

Have you thought about which route to go?

Some if not most of the gitlab client libraries I've used has taken the path of
fetching all results at once regardless of situation.
If done like that I guess It's easy to implment with a loop in the reqest function which just iterates and compares "last" with the current url until they matches.

I guess it would be better to have a synchronous version than nothing if that's faster to implement right now.

This is the one issue that actually makes emacs-gitlab useless right now.. I'm not interested in my closed tickets from a few years ago which are the only ones I see now.
EDIT: sorting has changed, so I see some new ones but only 20 of them.,

[marcinant]

This is probably something new in GitLab API.
It supports pagination:

    page (default: 1) - page number
    per_page (default: 20, max: 100) - number of items to list per page

This is why by default only first 20 projects/issues are on list. We should extend emacs-gitlab and add option to fetch all projects/milestones/issues or support pagination.

Question is how to solve this problem. Personally I don't use projects/isues list generated by emacs-gitlab.
As you can see I extended this project with some new stuff and I'm going to implement all remaining API.

Thing is how to resolve UI and how to handle projects/milestones/issues.
We could extend projects/issues list and try to implement something like gitlab interface in Emacs. However I think it's not good idea.
I import projects/milestones/issues to Org files. Then I can see all TODO's on my agenda and can clock them or change TODO state.

[thomasf]

It's only the link headers that reveals where the last page is though without resorting to some trial and "error" method. It's the same for all paged api resources except one or two which they missed to add (might be fixed)

[marcinant]

Well, I don't understand what are you going to achieve.

Could you please elaborate your workflow and what is the problem?

[thomasf]

All Gitlab API REST resource endpoints uses the link headers.. I don't understand what your comment about org files has any connection to this issue, however I am also only interested in the API side to eventually get my issues into git-commit-mode/auto-complete and org-mode.

[marcinant]

I thought that you are talking about lists generated by gitlab-show-projects' orgitlab-show-issues`.
Problem with pagination affects all implementations where you want to get all data.

Currently it's not possible to see all projects with gitlab-show-projects. you only get first 20.
Same if you want to import projects/issues etc. to emacs assoc list. You only get first 20 as well.

My comment about org files has this connection that you could use org-files to cache data from gitlab in nice format which can be parsed and filtered and you don't need to request gitlab every time.

[thomasf]

Alright..
In that case my suggestion is still to loop inside the get request* functions in gitlab-util that fetches all results, possibly in combination with defining an variable that makes it possible to limit the max results if desired.

[marcinant]

There is no other way. Gitlab API returns 20 elements by default and you need to extend this limit to 100 and loop if it's not enough.

There is an option to get all items however you need admin permissions on gitlab server.

So, the only way to improve performance is to cache results.

[thomasf]

The alternative to always looping for all results is to introduce paging into the elisp gitlab API but I cannot see much reason for this..

I don't think there is much of a performance issue in fetching even up to 10 pages of json.
The fetch loop should probably be implemented asynchronously under lexical scoping so that Emacs doesn't freeze up or or mixes up the results.
I think that the responsibility of caching should be on the consumer side (like the helm interface or whatever) and not a lower level thing.

[marcinant]

The alternative to always looping for all results is to introduce paging into the elisp gitlab API but I cannot see much reason for this..

I think that the responsibility of caching should be on the consumer side (like the helm interface or whatever) and not a lower level thing.

This is why paging should be supported by emacs gitlab api. With helm you can see limited amount of results. So, you could use paging to limit amount of data you fetch from gitlab.

Anyway I don't think that performance is a problem. Emacs gitlab api is based on request.el which can be synchronous or async.
However I think we should first implement all endpoints and functions to add/edit/delete items and provide some integration with tools such as org-mode or company/autocomplete to make things useful.

[thomasf]

I've just wrote some basic code to generate org tables for my issues in projects..
Was thinking about filtering..
I'm short on time so I did not try to debug why emacs-gitlab doesnt seem to do paging correctly..

As a work aroynd I'm using this function to remove duplicates

(defun my-gitlab-remove-duplicate-id (v)
  "TODO"
  (cl-remove-duplicates v
                     :test (lambda (x y)
                             (or (null y) (equal (assoc-default 'id x)
                                                 (assoc-default 'id y))))
                     :from-end t))

[thomasf]

Unrelated to this issue but.. I did this last week, Just the bare minimun that I need to easy my working day.. It's a one way sync just running a function to update all open buffer with queries in the properties drawer....

I will try to make time to work a bit more on it and release it later on.

At this time each property filter needs it's own function and filters only supports one entry so again, very basic.
It seems to me as I will write some cached filtering client on top on the basic client, we will have to see :)

[marcinant]

We definetly should coordinate our efforts.

I have got function fetching all project data from gitlab to local copy of repo.
I also have some functions to update gitlab status on org-mode hooks and some basic idea how to update org files on the fly as someone performs changes on gitlab server directly (via webhooks) look #22.

Unfortunately I don't have time to finish this and debug properly.

So, if we could prepare some nice org->gitlab interface together it could be great.

I'll try to upload my code as additional library on next weekend.

[nlamirault]

@thomasf great Org file !

[thomasf]

Yeah.. what I have is currently only what I'm pasting below... I have never work with org mode directly in this way before so I am still learning how to do that properly..

A priority for me is to extract all check boxes from every issue as sub check boxes into the checkbox item tree.

(require 'cl)
(require 'org)
(require 'gitlab)

(defun my-gitlab-remove-duplicate-id (v)
  (remove-duplicates v
                     :test (lambda (x y)
                             (or (null y) (equal (assoc-default 'id x)
                                                (assoc-default 'id y))))
                     :from-end t))

(defvar my-gitlab-projects-data nil)
(defun my-gitlab-projects ()
  (unless my-gitlab-projects-data
    (let ((p (gitlab-list-all-projects)) )
      (setq
       my-gitlab-projects-data
       (append (my-gitlab-remove-duplicate-id p) nil))))
  my-gitlab-projects-data)


(defun my-gitlab-project-by-id (project-id)
  (--first
   (equal project-id (assoc-default 'id it))
   (my-gitlab-projects)))

(defun my-gitlab-project (path-with-namespace)
  (--first
   (equal path-with-namespace (assoc-default 'path_with_namespace it))
   (my-gitlab-projects)))


(defun my-gitlab-project-id (project)
  "TODO"
  (assoc-default 'id project))

(defun my-gitlab-clear-cache ()
  (interactive)
  (setq my-gitlab-projects-data nil))

(defun my-gitlab-project-issues (project)
  "TODO"
  (my-gitlab-remove-duplicate-id
   (gitlab-list-all-project-issues
    (my-gitlab-project-id project))))

(defun my-gitlab-issues-filter-assignee (assignee issues)
  (delq nil
        (mapcar
         #'(lambda (issue)
             (when
                 (equal assignee
                        (assoc-default
                         'username
                         (or (assoc-default 'assignee issue)
                            '(username))))
               issue))
         issues)))

(defun my-gitlab-issues-filter-opened (issues)
  (delq nil
        (mapcar
         #'(lambda (issue)
             (when
                 (equal "opened"
                        (assoc-default 'state issue))
               issue))
         issues)))


(defun my-gitlab-issues-filter-milestone (iid issues)
  (delq nil
        (mapcar
         #'(lambda (issue)
             (let ((ms (assoc-default 'milestone issue)) )
               (if (and ms (equal iid (assoc-default 'iid  ms)))
                   issue
                 nil)))
         issues)))

(defun my-gitlab-issues-org-update ()
  (interactive)
  (mapc #'(lambda (buffer)
            (with-current-buffer buffer
              (when
                  (eq major-mode 'org-mode)

                (let ((case-fold-search nil))
                  (save-excursion
                    (save-restriction
                      (widen)
                      (goto-char (point-min))
                      (while (re-search-forward
                              (concat "^[ \t]*:gitlab_issues:[ \t]+.*$")
                              nil t)
                        (save-restriction
                          (org-narrow-to-subtree)
                          (delete-region  (org-end-of-meta-data-and-drawers) (org-end-of-subtree t t ))
                          (let* ((gitlab-project (org-entry-get-with-inheritance  "gitlab_issues"))
                                 (gitlab-assigned (org-entry-get-with-inheritance  "gitlab_assigned"))
                                 (gitlab-milestone (org-entry-get-with-inheritance  "gitlab_milestone"))
                                 (issues (my-gitlab-project-issues (my-gitlab-project gitlab-project))))
                            (when gitlab-assigned
                              (setq issues (my-gitlab-issues-filter-assignee gitlab-assigned issues)))
                            (when gitlab-milestone
                              (setq issues (my-gitlab-issues-filter-milestone (string-to-number gitlab-milestone) issues)))
                            (insert (my-gitlab-issues-org-tbl issues))
                            (org-update-checkbox-count)))
                        (org-end-of-subtree t t)
                        )))))))
        (buffer-list)))

(defun  my-gitlab-issues-org-tbl (issues)
  (with-temp-buffer
    (let ((in-subtree) )
      (mapc
       #'(lambda (issue)
           (let* ((project (my-gitlab-project-by-id (assoc-default 'project_id issue)))
                  (project-nspath (assoc-default 'path_with_namespace project))
                  (issue-iid (assoc-default 'iid issue))
                  (issue-state (assoc-default 'state issue))
                  (issue-title (assoc-default 'title issue))
                  (issue-assignee (assoc-default
                                   'username
                                   (or (assoc-default 'assignee issue)
                                      '(username ""))))
                  (issue-url (format "http://gitlab.hostname/%s/issues/%d" project-nspath issue-iid))
                  (issue-link (org-make-link-string issue-url (format "#%d" issue-iid))))
             (insert
              (cond
               ((string= issue-state "closed") "- [X] " )
               ((string= issue-state "opened") "- [ ] " ))
              " "
              ;; (if (string= issue-state "opened") "TODO"  "DONE")
              issue-link
              " "
              issue-title
              ;; " "
              ;; (if (string= issue-state "opened") ":open:"  ":closed:")
              )
             (end-of-line) (newline)
             (when issue-assignee
               (insert "       ")
               (insert "assigned: " issue-assignee)
               (end-of-line) (newline))

             (end-of-line) (newline)

             )) 
       issues))
    (buffer-string)))

(provide 'my-gitlab)

[nlamirault]

For now, i juste create a file like that :

* FOO
#+CATEGORY: Gitlab
** M1
*** TODO [[gitlab:/me/foo/issues/5][Issue 5]]
    DEADLINE: <2015-12-31>
*** TODO [[gitlab:/me/foo/issues/8][Issue 8]]
    DEADLINE: <2015-12-31>
** M2
*** TODO [[gitlab:/me/foo/issues/12][Issue 12]]
    DEADLINE: <2016-01-31>
*** TODO [[gitlab:/me/foo/issues/7][Issue 7]]
    DEADLINE: <2016-01-31>
...

#+LINK: gitlab  https://gitlab.xxxxxxxx

with this code :

(require 'cl)
(require 'org)
(require 'gitlab)


(defun gitlab-issue-status-to-org (issue)
  (cond ((string= (assoc-default 'state issue) "opened")
         "TODO")
        ((string= (assoc-default 'state issue) "active")
         "NEXT")
        ((string= (assoc-default 'state issue) "closed")
         "DONE")
        (t (assoc-default 'state issue))))


(defun gitlab-to-org ()
  (interactive)
  (with-temp-buffer
    (mapc (lambda (project)
            (insert "* " (assoc-default 'name project))
            (end-of-line) (newline)
            (insert "#+CATEGORY: Gitlab")
            (end-of-line) (newline)
            (mapc (lambda (milestone)
                    (insert "** " (assoc-default 'title milestone))
                    (end-of-line) (newline)
                    (mapc (lambda (issue)
                            (insert "*** "
                                    (gitlab-issue-status-to-org issue)
                                    " "
                                    "[[gitlab:/"
                                    (assoc-default 'path_with_namespace project)
                                    "/issues/"
                                    (number-to-string (assoc-default 'id issue))
                                    "]["
                                    (assoc-default 'title issue)
                                    "]]")
                            (end-of-line) (newline)
                            (insert "    DEADLINE: <"
                                    (assoc-default 'due_date milestone)
                                    ">")
                            (end-of-line) (newline))
                          (gitlab-get-milestone-issues (assoc-default 'id project)
                                                       (assoc-default 'id milestone))))
                  (gitlab-list-project-milestones (assoc-default 'id project)))
            (insert "** ALL")
            (end-of-line) (newline)
            (mapc (lambda (issue)
                    (unless (assoc-default 'milestone issue)
                      (insert "** "
                              (gitlab-issue-status-to-org issue)
                              " "
                              (assoc-default 'title issue))
                      (end-of-line) (newline)))
                  (gitlab-list-all-project-issues (assoc-default 'id project))))
          (gitlab-list-projects))
    (end-of-line) (newline)
    (end-of-line) (newline)
    (insert "#+LINK: gitlab  " gitlab-host)
    (buffer-string)))