[Bug][Config-ui] Adding data scope succeeds but the scope is never listed

[renflo]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

Ia have a gitlab connection which works for many projects using it. I am creating a new devlake project but I can't manage to add a new data scope to it. Each time I go through the wizard to add the data scope, it terminates successfully (message under) but when I get back to the connection at https://<devlake-ui-host>/connections/gitlab/2 it does not list the new data scope, which implicitly means that I can't configure this new project as I can't assign it a data scope, which is a blocker for me.

What do you expect to happen

I am expecting to find the new data scope in the list of scopes so I can select and use it.

How to reproduce

1. Create a gitlab connection
2. Add datascope to it
3. Verify that you get the message confirming their creation
4. Verify that you find the created data scopes and are able to use/select them

Make sure to do 3-5 more than 50 times, I suspect there could be a limit somewhere, see next section..

I could reproduce this behavior with several repositories on my gitlab connection, so it does not seem to be related to the repositories themselves.

Anything else

I notice that I have 50 datascopes listed in this connection. Could there be a limitation in your code which only returns 50 datascope objects? I find it peculiar that it stops at such a round number.

I find no suspicious message in config-ui's logs.

Version

0.18.0beta4

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[d4x1]

Wired. Does the backend have any logs?

[renflo]

Wired. Does the backend have any logs?

I'll recheck all running components, and get back to you.

[renflo]

So I tried to re-add and see that in the lake container

[GIN] 2023/08/11 - 07:00:26 | 200 |  1.742605047s |  84.52.xxx.yyy | GET      "/plugins/gitlab/connections/2/remote-scopes"
[GIN] 2023/08/11 - 07:00:32 | 200 |   79.288288ms |       127.0.0.6 | GET      "/blueprints"
[GIN] 2023/08/11 - 07:00:40 | 200 |  2.124177654s |   84.52.xxx.yyy | GET      "/plugins/gitlab/connections/2/remote-scopes?groupId=group:7316105"
[GIN] 2023/08/11 - 07:00:42 | 200 |   61.769958ms |       127.0.0.6 | GET      "/blueprints"
[GIN] 2023/08/11 - 07:00:52 | 200 |   67.220508ms |       127.0.0.6 | GET      "/blueprints"
[GIN] 2023/08/11 - 07:00:54 | 200 |  1.809503096s |   84.52.xxx.yyy | GET      "/plugins/gitlab/connections/2/remote-scopes?groupId=group:68606318"
[GIN] 2023/08/11 - 07:01:02 | 200 |   67.745858ms |       127.0.0.6 | GET      "/blueprints"
[GIN] 2023/08/11 - 07:01:04 | 200 |  2.859834997s |   84.52.xxx.yyy | GET      "/plugins/gitlab/connections/2/remote-scopes?groupId=group:13893691"
[GIN] 2023/08/11 - 07:01:10 | 200 |  2.645127344s |   84.52.xxx.yyy| GET      "/plugins/gitlab/connections/2/remote-scopes?groupId=group:57323903"

And the config-ui container logs:

127.0.0.6 - user [11/Aug/2023:07:00:24 +0000] "GET /assets/icon-496462d1.svg HTTP/1.1" 200 1601 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/114.0" "84.52.xxx.yyy, 127.0.0.6"
127.0.0.6 - user [11/Aug/2023:07:00:26 +0000] "GET /api/plugins/gitlab/connections/2/remote-scopes HTTP/1.1" 200 131 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/114.0" "84.52.xxx.yyy, 127.0.0.6"
127.0.0.6 - user [11/Aug/2023:07:00:40 +0000] "GET /api/plugins/gitlab/connections/2/remote-scopes?groupId=group:7316105 HTTP/1.1" 200 7060 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/114.0" "84.52.xxx.yyy, 127.0.0.6"
127.0.0.6 - user [11/Aug/2023:07:00:54 +0000] "GET /api/plugins/gitlab/connections/2/remote-scopes?groupId=group:68606318 HTTP/1.1" 200 3538 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/114.0" "84.52.xxx.yyy, 127.0.0.6"
127.0.0.6 - user [11/Aug/2023:07:01:04 +0000] "GET /api/plugins/gitlab/connections/2/remote-scopes?groupId=group:13893691 HTTP/1.1" 200 2985 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/114.0" "84.52.xxx.yyy, 127.0.0.6"
127.0.0.6 - user [11/Aug/2023:07:01:10 +0000] "GET /api/plugins/gitlab/connections/2/remote-scopes?groupId=group:57323903 HTTP/1.1" 200 3014 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/114.0" "84.52.xxx.yyy, 127.0.0.6"
127.0.0.6 - user [11/Aug/2023:07:01:14 +0000] "PUT /api/plugins/gitlab/connections/2/scopes HTTP/1.1" 200 2045 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/114.0" "84.52.xxx.yyy, 127.0.0.6"
127.0.0.6 - user [11/Aug/2023:07:01:14 +0000] "GET /api/plugins/gitlab/connections/2/scopes?blueprints=true HTTP/1.1" 200 1142915 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/114.0" "84.52.xxx.yyy, 127.0.0.6"

I imagine none of that is helping.

[nathan-nicholson]

I'm actually experiencing the exact same issue on the GitHub collector on the same version of DevLake (0.18.0-beta4). Looks suspiciously like a pagination limit or something, but I haven't looked through source to confirm yet.

[klesh]

@nathan-nicholson @renflo I think you are correct, it is the pagination limit

incubator-devlake/backend/helpers/pluginhelper/api/scope_db_helper.go

Line 123 in 5046993

err := s.db.All(&scopes, query, dal.Limit(limit), dal.Offset(offset))

[klesh]

@mintsweet we should add support for pagination for the scope lists.

[renflo]

@klesh that would be great, I understand the project is in incubation and it's perhaps not what you want to focus on solving now but for me that is a blocker.

If implementing "the correct/long term solution" is to cumbersome perhaps going KISS would work: remove the limit totally or make it arbitrarily high (10000) for example.

[klesh]

@renflo Yup, I agree with you.
However, pagination support is long overdue for nearly 2 years 🥲 and I believe it is time to deal with it once and for all.

[klesh]

Hi, @renflo , After some discussion, we decided to support pagination on v0.19 and remove it from v0.18 because cherry-picking is quite hard between them.