API paginater problem

[debMan]

• Gitea version (or commit ref): 1.9.4
• Git version: 2.17.1
• Operating system: ubuntu 18.04
• Database:
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No
  • Not relevant

Description

Hello everyone. Glad to be here.
I did not have experience in go lang. I had a workaround on the codes but could not fix this,

Summary

During the import of Gitea repositories into GitLab, some issues may be skipped over due to query string parameters changing during API pagination requests. Specialty in the last pages.

Steps to reproduce

You can reface this when importing repos to the GitLab. On some repositories with many numbers of issues and PRs, the pagination API responses next and prev page numbers incorrect

Logs

Here i'\ insert the logs which local Gitea server shows when importing repositories from Gitea to the GitLab:

10.242.12.2 - - [26/Nov/2019:07:45:18 +0000] "GET /api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=2&per_page=100&sort=created&state=all HTTP/1.1" 200 9886 "-" "Octokit Ruby Gem 4.9.0"
10.242.12.2 - - [26/Nov/2019:07:45:19 +0000] "GET /api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=3&per_page=100&sort=created&state=all HTTP/1.1" 200 13483 "-" "Octokit Ruby Gem 4.9.0"
10.242.12.2 - - [26/Nov/2019:07:45:20 +0000] "GET /api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=4&per_page=100&sort=created&state=all HTTP/1.1" 200 9115 "-" "Octokit Ruby Gem 4.9.0"
10.242.12.2 - - [26/Nov/2019:07:45:21 +0000] "GET /api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=5&per_page=100&sort=created&state=all HTTP/1.1" 200 8913 "-" "Octokit Ruby Gem 4.9.0"
10.242.12.2 - - [26/Nov/2019:07:45:23 +0000] "GET /api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=6&per_page=100&sort=created&state=all HTTP/1.1" 200 8991 "-" "Octokit Ruby Gem 4.9.0"
10.242.12.2 - - [26/Nov/2019:07:45:24 +0000] "GET /api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=7&per_page=100&sort=created&state=all HTTP/1.1" 200 8268 "-" "Octokit Ruby Gem 4.9.0"

The problem is that more pages still available but Gitea responses the last page is page 7. To show more details, let me request a currect scenario on page 6 something like:

request

https://git.ourOrganazation.com/api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=6&per_page=100&sort=created&state=all

response header

<https://git.ourOrganazation.com/api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=7&per_page=100&sort=created&state=all&state=all>; rel="next",
<https://git.ourOrganazation.com/api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=7&per_page=100&sort=created&state=all&state=all>; rel="last",
<https://git.ourOrganazation.com/api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=1&per_page=100&sort=created&state=all&state=all>; rel="first",
<https://git.ourOrganazation.com/api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=5&per_page=100&sort=created&state=all&state=all>; rel="prev"

But the last page is not page 7, let me request an incorrect scenario on page 7 and 8 something like:

request page 7

https://git.ourOrganazation.com/api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=7&per_page=100&sort=created&state=all

response header page 7

<https://git.ourOrganazation.com/api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=1&per_page=100&sort=created&state=all&state=all>; rel="first",
<https://git.ourOrganazation.com/api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=6&per_page=100&sort=created&state=all&state=all>; rel="prev"

request page 8

https://git.ourOrganazation.com/api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=8&per_page=100&sort=created&state=all

response header page 8

<https://git.ourOrganazation.com/api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=1&per_page=100&sort=created&state=all&state=all>; rel="first",
<https://git.ourOrganazation.com/api/v1/repos/ourOrganazation/ourRepo/issues?direction=asc&page=6&per_page=100&sort=created&state=all&state=all>; rel="prev"

but we can see still page 8 is valid and has data, the response body of page 8:

[
    {
        "id": 6043,
        "url": "https://git.ourOrganazation.com/api/v1/repos/ourOrganazation/ourRepo/issues/15",
        "number": 15,
        "user": {
            "id": 1,
            "login": "admin",
            "full_name": "admin",
            "email": "***@***.com",
            "avatar_url": "https://secure.gravatar.com/avatar/***",
            "language": "en-US",
            "is_admin": true,
            "last_login": "2019-11-25T06:40:56Z",
            "created": "2019-08-01T13:23:17Z",
            "username": "admin"
        },
        "title": "Set limits",
 ....

PS:

• I know some query strings like per_page, direction, etc. are not handled by Gittea, but this does not result in a problem. The requests made by GitLab.
• I know the problem is result of reponsed header link field which set the next,prev, first and last pages.

[hilariocoelho]

#9452 should close this

Actually I don’t think it closes. I’ve analyzed branch master and I’m not sure my PR fixed that issue. I think it is other bug, but I’m not sure

[6543]

o title was litle bit misleading .. you are right

[stale]

This issue has been automatically marked as stale because it has not had recent activity. I am here to help clear issues left open even if solved or waiting for more insight. This issue will be closed if no further activity occurs during the next 2 weeks. If the issue is still valid just add a comment to keep it alive. Thank you for your contributions.

[6543]

ping

[6543]

current status of this?

[6543]

@debMan does this happen on current v1.11.3 too?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. I am here to help clear issues left open even if solved or waiting for more insight. This issue will be closed if no further activity occurs during the next 2 weeks. If the issue is still valid just add a comment to keep it alive. Thank you for your contributions.

[stale]

This issue has been automatically closed because of inactivity. You can re-open it if needed.