🚀 Feature: scope Gitlab Org Data integration to a single group

[jamieklassen]

🔖 Feature description

The group field for a GitlabProviderConfig is described as

backstage/plugins/catalog-backend-module-gitlab/config.d.ts

Lines 32 to 36 in e9e5da2

	/**
	* (Optional) Gitlab's group[/subgroup] where the discovery is done.
	* If not defined the whole project will be scanned.
	*/
	group?: string;

and

backstage/docs/integrations/gitlab/discovery.md

Line 25 in e9e5da2

group: example-group # Optional. Group and subgroup (if needed) to look for repositories. If not present the whole project will be scanned

This is consistent with the behaviour of the GitlabDiscoveryEntityProvider, which uses the parameter to filter projects:

backstage/plugins/catalog-backend-module-gitlab/src/providers/GitlabDiscoveryEntityProvider.ts

Lines 155 to 162 in e9e5da2

	const projects = paginated<GitLabProject>(
	options => client.listProjects(options),
	{
	group: this.config.group,
	page: 1,
	per_page: 50,
	},
	);

Since the GitlabOrgDiscoveryEntityProvider also consumes a GitlabProviderConfig, it could interpret the same parameter in a corresponding fashion regarding groups and users:

• the only groups ingested are the referenced group itself and its descendants.
• the only users ingested are direct members of the referenced group or its descendants.

🎤 Context

Narrowing the set of users and groups to ingest is helpful when your organization is using a GitLab instance shared with other software projects that are not tracked in Backstage. Also, if your GitLab instance contains a lot of data, it can be helpful to narrow the scope of ingestion to get faster feedback while testing and iterating on your provider config.

✌️ Possible Implementation

In #16023 I mention a GraphQL query that can fetch a group and its descendant groups, as well as direct memberships at each level of the hierarchy. That's a pretty good fit for this issue too!

👀 Have you spent some time to check if this feature request has been raised before?

• I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• I have read the Code of Conduct

Are you willing to submit PR?

Yes I am willing to submit a PR!

[benjdlambert]

Thanks for creating this and #16023. Think they both seem reasonable! 🙏

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[sbarrypoppulo]

Still needs to be implemented

[cthtrifork]

Anyone up for this? Right now we are using the public gitlab.com and backstage tries to list/discover all users which are hundreds of thousands..

[sbarrypoppulo]

My team is currently working on this. Hoping to have some changes to review and collect feedback in the coming days

[matteosilv]

Hi, just wanted to comment that this stuff is risky as in GitLab self-hosted it's reasonable that you want all users (of the whole instance) but just a subgroup for teams, and that's why I worked on a merged MR in the past to allow fetch only subgroups of a specific group:

        group: org/teams # Optional. Must not end with slash. Accepts only groups under the provided path (which will be stripped)

Instead in the SaaS instance you still want the second option as your teams description will definitely be under some subgroup, but you only want the user that are member of your root level group, so we should probably add an option that tell us which is the root level group.

[matteosilv]

@sbarrypoppulo if you need help I'd be happy to work with you on this

[sbarrypoppulo]

@matteosilv Yes, having to support slightly different logic for Self-Managed GitLab and GitLab.com is a bit of a challenge. We've handled this in our implementation like so:

• For GitLab Self-Managed: Continue to use the /users and /groups endpoints (group parameter is optional)
• For GitLab.com: Use GraphQL queries to fetch group and it's descendant groups and fetch the direct members of the groups (group is required for GitLab.com)

We're fairly close to opening a PR for review, so would appreciate any feedback and testing of the proposed changes! Our solution is potentially geared towards our Orgs particular group and membership structure, but will hopefully provide something that can be built upon.

[matteosilv]

For example with your current implementation you seem to be breaking the assumption I made with my MR:

In my case I have my root level group that is my-company and my teams are under the subgroup my-company/teams, so when I list user I'd like them t be listed under the root level group and not under my-company/teams, so I think we should change this line:

  users = (await client.listSaasUsers(this.config.group)).items;

and pass the root level group, stripping out any subgroup.

What do you think?

[matteosilv]

I commented in your fork. I can help and test it on my instance if needed

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[vinzscam]

Closing this as resolved by #18889