Feature/gitlab integration #11105

e-stadnik · 2023-07-16T14:12:04Z

Fixes #11104

Proposed changes

Added build server integration for Gitlab pipelines

Screenshots

Before

After

Test methodology

Unit tests for adapter logic are included
Manual testing on private commercial solution on self-hosted Gitlab instance
Manual testing on open-source repo: https://gitlab.com/wireshark/wireshark

Test environment(s)

GIT 2.33.0.windows.2
Windows 10 22H2 19045.3208

Merge strategy

To be decided later.

I agree that the maintainer squash merge this PR (if the commit message is clear).

✒️ I contribute this code under The Developer Certificate of Origin.

gerhardol · 2023-07-17T20:12:47Z

Can this be tried with public gitlab.com?
What to configure?
Can you please add this to source/settings.rst?

gerhardol

textboxes are wider than the the minimal width
very brief review but did not react to anything

GitExtensions.sln

Plugins/BuildServerIntegration/GitlabIntegration/ApiClient/GitlabApiClient.cs

RussKie · 2023-07-18T10:02:49Z

Thank you for the contribution but I am really not excited about adding another plugin in to the main repo. My lack of excitement stems from several things: 1. Ownership - the Core team will essentially be responsible for maintaining the plugin, triaging issues and fixing bugs after a short while. 2. Release cadence - we're releasing the app few times a year, so any issues with the plugin won't get addressed in timely manner. E.g., GitLab changed its API, or bugs discovered in the plugin or few features requested. It'd be desirable if the plugin was released as a standalone plugin which users can install via the Plugin Manager.

e-stadnik · 2023-07-18T18:38:38Z

Thank you for feedback and review tips.

As Gitlab mentioned in docs all significant API changes come major GitLab releases only. It should be very minor chance to receive unexpected failtures.

The logic here is quite simple and similar to other integrations included in main repo.

Can we discuss durability requirements which will make possible to merge it in product?
Because Gitlab.com allows token free access for public repos, I'm about to add integration tests.

gerhardol · 2023-07-19T21:18:51Z

The logic here is quite simple and similar to other integrations included in main repo.

Can we discuss durability requirements which will make possible to merge it in product? Because Gitlab.com allows token free access for public repos, I'm about to add integration tests.

The difference is if any maintainer use the feature or not. (I have no info if someone uses TeamCity or if the integration exists). It is quite frustrating to get issue reports for features you know nothing about and cannot test.

Build server integration are quite special plugins, deeply integrated to GE so there are arguments to keep them in as well, then general cleanups are done as well.
@RussKie takes the decision.

gerhardol

Still a visual:

Can you please describe how to use?
Can it be used with gitlab.com?

gerhardol · 2023-07-26T19:38:00Z

Plugins/BuildServerIntegration/GitlabIntegration/ApiClient/Models/GitlabPipeline.cs

+{
+    public class GitlabPipeline
+    {
+        public int id { get; set; }


Uppercase, IDE10006
https://learn.microsoft.com/sv-se/dotnet/fundamentals/code-analysis/style-rules/naming-rules

gerhardol · 2023-07-26T19:41:33Z

Plugins/BuildServerIntegration/GitlabIntegration/ApiClient/GitlabApiClientFactory.cs

@@ -0,0 +1,21 @@
+using System;


Usings are not needed

gerhardol · 2023-07-26T19:53:45Z

Plugins/BuildServerIntegration/GitlabIntegration/GitlabAdapter.cs

+            Validates.NotNull(_apiClient);
+
+            PagedResponse<GitlabPipeline> firstPage = await _apiClient.GetPipelinesAsync(sinceDate, running, 1);
+            firstPage.Items.ForEach(item => ProcessLoadedBuild(item, observer));


RussKie will ask this to changed to a normal loop

e-stadnik · 2023-07-26T20:31:59Z

Can you please describe how to use?
Can it be used with gitlab.com?

Sure, I tested on wireshark repo

For settings use Instance URL https://gitlab.com/
And project id 7898047
Token field can be left empty because it's a public project.

For private projects token can be obtained according to this guide

pmiossec · 2023-07-26T21:22:31Z

Some potential improvements:

Can it be used with gitlab.com?

Sure, I tested on wireshark repo

So maybe it could be a good idea to set "https://gitlab.com" by default

For settings use Instance URL https://gitlab.com/ And project id 7898047 Token field can be left empty because it's a public project.

If projectId is mandatory but not the token, for me it makes more sense to put the projectId field before the token one.

For private projects token can be obtained according to this guide

I would at least put a link to this documentation. But even better looking at the documentation, you can build a link open the url (using the server url) to let the user easily create a token. From the doc https://gitlab.example.com/-/profile/personal_access_tokens?name=GitExtensions+Access+token&scopes=api,read_user,read_registry...
It is already done for the AzureDevOps Ci (TokenManagementLink control in SettingsUserControl.cs)

e-stadnik · 2023-07-27T09:28:18Z

Philippe, thanks for your feedback!

I'm going to fill URL and project ID right from remotes.
Along with a link to obtain a token if necessary.

pmiossec · 2023-07-27T09:45:51Z

I'm going to fill URL and project ID right from remotes.

Great!
But is it possible?
I don't know GitLab works but previously you said that you need the projectId 7898047 but you don't have it from the remote url ( https://gitlab.com/wireshark/wireshark.git).... There is a open api to get it?

e-stadnik · 2023-07-27T09:52:27Z

But is it possible?

I'll try to lookup this way https://docs.gitlab.com/ee/api/projects.html#search-for-projects-by-name
When url_to_repo matches with remote it will be exact project id

pmiossec · 2023-07-27T10:15:37Z

It's a little strange but with the request https://gitlab.com/api/v4/projects?search=wireshark&sort=asc I don't get the wireshark project in the list returned. We need a token to have the full list?

Note: Maybe the field path_with_namespace is better (but you will have to extract it from ssh or http urls...)

RussKie · 2023-07-27T10:15:41Z

I'm away on holidays without an access to a computer (digital detox 😆). So it may be some time before I can check it out.

pmiossec · 2023-07-27T10:16:55Z

digital detox 😆

Mine starting in 2 days 😆

e-stadnik · 2023-07-27T10:45:22Z

It's a little strange but with the request https://gitlab.com/api/v4/projects?search=wireshark&sort=asc I don't get the wireshark project in the list returned. We need a token to have the full list?

You see paged response. The target one can be somewhere deep.
But I've found other issue, that project can be renamed without changing remote path.

pmiossec · 2023-07-27T11:32:03Z

But I've found other issue, that project can be renamed without changing remote path.

Yes, so don't add too much complexity for something that is only made once and takes 2 minutes to do.
Even me that don't know gitlab found it in 10s...

e-stadnik · 2023-07-27T15:39:04Z

But I've found other issue, that project can be renamed without changing remote path.

Yes, so don't add too much complexity for something that is only made once and takes 2 minutes to do. Even me that don't know gitlab found it in 10s...

The solutions was easier
https://gitlab.com/api/v4/projects/wireshark%2Fwireshark returns whole project info.

gerhardol · 2023-08-07T14:50:41Z

The ProjectId seems to be set for only one of the remotes. In my case I have "origin" with my changes and "upstream". The status is only shown for "origin".
AppVeyor and Jenkins at least allows several urls.

In addition, ProjectId is normally empty when opening the settings. I would like to say that the value is not read too, but it may be that I only got the build for origin but were looking for upstream builds.

e-stadnik · 2023-08-07T20:34:26Z

I've never worked in multiple remotes with configured CI/CD on different servers.
But think Gitlab is mostly designed to operate with stages in a single project.
The stages are equivalent of build configurations in TeamCity terminology.

Maybe I miss some important case, but anyway to support pipeline statuses from multiple origins we should allow to set several instance URLs.

About ProjectId loading, it was bug, fixed :)

gerhardol · 2023-08-07T22:05:08Z

Saving ProjectId seem to work for me now.

My question was about multiple owners on the same server, origin and upstream.
It is at least possible to switch between them now, so you can see builds from each even if it is hard to switch to upstream.
Not necessarily something to handle now.

I actually use multiple servers too. Not just different URL but different implementations for different suppliers: Gitlab to BitBucket to GitHub and back again.
But that is a separate setup, I use unique clones to handle this. Can unfortunately not use WorkTrees, GE does not store unique data per repo (not a big problem, build server is only important for me in BB). No private clones in BB, but multiple 'users' in GL.

@RussKie How to proceed here? Include this now and separate later?
If the plugin fails, it will have to wait for next normal update...

spdr870 · 2023-08-08T10:37:45Z

@e-stadnik
I use the gitlab community edition, self-hosted. I tried this plugin. The status is visible. The biggest issue I have is the performance. The repository is over 200.000 commits, and the UI freezes until the status is fetched for, I assume, all commits? To make it workable, I changed the code only to load the first page. Not sure how other plugins work, but a limit would be needed for large repositories. And maybe check for cancellationToken.IsCancellationRequested when loading pages?

Also; the build report tab page couldn't load the content since the inline browser is not supported by gitlab. Not a problem for me, the link to open it in an external browser works.

e-stadnik · 2023-08-08T11:02:39Z

@spdr870, Thanks a lot for you feedback!

Can you see page numbers on pipelines web page for you project?
From some threshold plugin loading all pages one by one, because total pages disappears from API as well. (like in wireshark repo). I didn't face with performance issues here.

If value for you project is below, plugin trying to request pages in parallel. In that case I'll try to change scheduler logic and it should helps with performance.

As constant solution I think to cache loaded entities on disk because API will return updated entities.

spdr870 · 2023-08-08T13:11:47Z

Can you see page numbers on pipelines web page for you project?

No pagenumbers, just Next. I manually altered the URL to find the number of pipelines, I could go to page 3342:
http://gitlab/Development/*********/-/pipelines?page=3342&scope=all

We run ~80 pipelines per day

gerhardol · 2023-08-08T15:48:37Z

The biggest issue I have is the performance. The repository is over 200.000 commits, and the UI freezes until the status is fetched for, I assume, all commits? To make it workable, I changed the code only to load the first page. Not sure how other plugins work, but a limit would be needed for large repositories. And maybe check for cancellationToken.IsCancellationRequested when loading pages?

Build status should be fetched and updated in the background (it is for at least Jenkins and AppVeyor).

Also; the build report tab page couldn't load the content since the inline browser is not supported by gitlab. Not a problem for me, the link to open it in an external browser works.

The tab should be disabled by default for all build servers.
The WebBrowser control could be replaced with WebView2 (https://learn.microsoft.com/en-us/microsoft-edge/webview2/) (only Win10 and later) or some Chromium control (adds to size), but I propose to just change the defaults.

…els/GitlabPipeline.cs Co-authored-by: Gerhard Olsson <6248932+gerhardol@users.noreply.github.com>

e-stadnik · 2023-08-31T19:08:02Z

I added BuildServer.Gitlab.PagesLimit settings but not sure about showing it in UI
It will appear in GitExtensions.settings file with default value (0 - no limitation) after save action in Gitlab settings page.
@spdr870 can you please check performance now with 10 or 50 pages?

e-stadnik · 2023-09-18T18:36:56Z

Can this be tried with public gitlab.com? What to configure? Can you please add this to source/settings.rst?

UI fields documentation added here: gitextensions/GitExtensionsDoc#148

@gerhardol sorry for force pushed commit, didn't expect that it'll break review comments. Hope everything will be fine for now.

gerhardol · 2023-09-19T22:04:17Z

@RussKie I suggest that this plugin is included. Separating buildserver plugins may be done later.

RussKie · 2023-09-20T02:59:49Z

Plugins/BuildServerIntegration/GitlabIntegration/Settings/GitlabSettingsUserControl.Designer.cs

+            GetProjectIdLink.Location = new Point(82, 54);
+            GetProjectIdLink.Name = "GetProjectIdLink";
+            GetProjectIdLink.Size = new Size(749, 15);
+            GetProjectIdLink.TabIndex = 9;


Could you please confirm the tab order is logically sequenced?

RussKie · 2023-09-20T10:51:53Z

Here's a test artifact: https://ci.appveyor.com/project/gitextensions/gitextensions/builds/48079386/artifacts
Could you please take it for a spin, install it and see if everything is working as you expect? If everything is good, we'll merge this.
Thank you.

e-stadnik · 2023-09-20T11:16:36Z

I'd been testing build 4.2.0.17315 for a last 2 weeks on a private repo.

On recent build checked tabIndex and pages loading limitation. Everything looks fine.

gerhardol · 2023-09-20T11:50:27Z

I have been using this pr for some time too, but my interaction with gitlab is quite limited, so it does not push any limit.
No proper review, but it looks good.

RussKie · 2023-09-20T12:37:59Z

Thank you

ghost assigned e-stadnik Jul 16, 2023

e-stadnik force-pushed the feature/GitlabIntegration branch from 26dcc7c to 5690781 Compare July 16, 2023 14:14

e-stadnik marked this pull request as ready for review July 16, 2023 16:22

e-stadnik mentioned this pull request Jul 16, 2023

Gitlab pipelines build plugin #11104

Closed

gerhardol reviewed Jul 17, 2023

View reviewed changes

GitExtensions.sln Outdated Show resolved Hide resolved

Plugins/BuildServerIntegration/GitlabIntegration/ApiClient/GitlabApiClient.cs Outdated Show resolved Hide resolved

e-stadnik marked this pull request as draft July 23, 2023 08:56

gerhardol reviewed Jul 26, 2023

View reviewed changes

e-stadnik force-pushed the feature/GitlabIntegration branch from 562e7b0 to 11a1caf Compare August 6, 2023 08:37

e-stadnik and others added 9 commits August 31, 2023 21:44

Fix build order (gitextensions#11104)

165da39

Code review fixes (gitextensions#11104)

98491dc

settings control improvements (gitextensions#11104)

b0c5739

update GitlabRemoteParserTests (gitextensions#11104)

229e5e6

Fixed project id loading from settings (gitextensions#11104)

8ab6f32

Update Plugins/BuildServerIntegration/GitlabIntegration/ApiClient/Mod…

57ece20

…els/GitlabPipeline.cs Co-authored-by: Gerhard Olsson <6248932+gerhardol@users.noreply.github.com>

Added cancellation token to API client (gitextensions#11104)

2a49b4e

Fix Gitlab adapter unit tests (gitextensions#11104)

019dcc8

Added pages limit setting for Gitlab adapter (gitextensions#11104)

571c8ad

e-stadnik force-pushed the feature/GitlabIntegration branch from 0f26a2e to 571c8ad Compare August 31, 2023 18:44

Set default Gitlab pipelines pages limit as 0 (gitextensions#11104)

ec8f687

e-stadnik marked this pull request as ready for review August 31, 2023 21:30

e-stadnik mentioned this pull request Sep 18, 2023

Added doc for Gitlab pipelines integration gitextensions/GitExtensionsDoc#148

Merged

RussKie reviewed Sep 20, 2023

View reviewed changes

RussKie added the 📭 needs: author feedback More info/confirmation awaited from OP; issues typically get closed after 30 days of inactivity label Sep 20, 2023

Adjust tab indexes for Gitlab Settings control(gitextensions#11104)

4f8b0ee

ghost removed the 📭 needs: author feedback More info/confirmation awaited from OP; issues typically get closed after 30 days of inactivity label Sep 20, 2023

RussKie added the 📭 needs: author feedback More info/confirmation awaited from OP; issues typically get closed after 30 days of inactivity label Sep 20, 2023

ghost removed the 📭 needs: author feedback More info/confirmation awaited from OP; issues typically get closed after 30 days of inactivity label Sep 20, 2023

RussKie approved these changes Sep 20, 2023

View reviewed changes

RussKie merged commit cad0513 into gitextensions:master Sep 20, 2023
3 checks passed

ghost added this to the vNext milestone Sep 20, 2023

RussKie added the release-notes-worthy label Sep 20, 2023

Feature/gitlab integration #11105

Feature/gitlab integration #11105

Conversation

e-stadnik commented Jul 16, 2023 • edited

Proposed changes

Screenshots

Before

After

Test methodology

Test environment(s)

Merge strategy

gerhardol commented Jul 17, 2023

gerhardol left a comment

Choose a reason for hiding this comment

RussKie commented Jul 18, 2023 via email • edited by ghost

e-stadnik commented Jul 18, 2023

gerhardol commented Jul 19, 2023

gerhardol left a comment

Choose a reason for hiding this comment

gerhardol Jul 26, 2023

Choose a reason for hiding this comment

gerhardol Jul 26, 2023

Choose a reason for hiding this comment

gerhardol Jul 26, 2023

Choose a reason for hiding this comment

e-stadnik commented Jul 26, 2023

pmiossec commented Jul 26, 2023

e-stadnik commented Jul 27, 2023

pmiossec commented Jul 27, 2023

e-stadnik commented Jul 27, 2023

pmiossec commented Jul 27, 2023

RussKie commented Jul 27, 2023 via email • edited by ghost

pmiossec commented Jul 27, 2023

e-stadnik commented Jul 27, 2023

pmiossec commented Jul 27, 2023

e-stadnik commented Jul 27, 2023

gerhardol commented Aug 7, 2023

e-stadnik commented Aug 7, 2023 • edited

gerhardol commented Aug 7, 2023

spdr870 commented Aug 8, 2023

e-stadnik commented Aug 8, 2023

spdr870 commented Aug 8, 2023 • edited

gerhardol commented Aug 8, 2023

e-stadnik commented Aug 31, 2023

e-stadnik commented Sep 18, 2023 • edited

gerhardol commented Sep 19, 2023

RussKie Sep 20, 2023

Choose a reason for hiding this comment

RussKie commented Sep 20, 2023

e-stadnik commented Sep 20, 2023

gerhardol commented Sep 20, 2023

RussKie commented Sep 20, 2023

e-stadnik commented Jul 16, 2023 •

edited

RussKie commented Jul 18, 2023 via email •

edited by ghost

RussKie commented Jul 27, 2023 via email •

edited by ghost

e-stadnik commented Aug 7, 2023 •

edited

spdr870 commented Aug 8, 2023 •

edited

e-stadnik commented Sep 18, 2023 •

edited