Gitlab Discovery Processor should automatically add the project-id if not present in catalog-info.yaml / backstage.yaml

[chaughawout]

Feature Suggestion

The @loblaw-sre/backstage-plugin-gitlab depends on the gitlab project-id to be added to the entity in a specific location. It would be great if the discovery processor could automatically populate that id into that location so that users don't have to specify it in their yaml files. If the user does specify a project-id in the yaml file, the plugin could skip the attempt to populate it so that it doesn't override the value set by the user.

Possible Implementation

I don't have context into the discovery processor to provide implementation suggestions, but I know there might be alternatives to the gitlab discovery processor to populate those entities, and it would be great if this same feature could be implemented by them as well.

I believe these changes would live in this plugin:
@backstage/plugin-catalog-backend-module-gitlab

Context

• Entity discovery should automatically popualte project id in catalog-info.yaml/backstage.yaml loblaw-sre/backstage-plugin-gitlab#154
• Step 5 in the Setup Guide outlines the location in the yaml

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[freben]

If I understand the situation correctly, the discovery processor does have knowledge of what the project ID is, but later down the line when the UrlReader wants to read the contents of the generated Location entity, this knowledge is lost because it's not encoded in the URLs. Is that a correct assessment?

If so, then yeah, something needs to be added that lets this contextual information be passed down through the layers. Not sure what the best solution would be, off the top of my head, but suggestions are welcome.

[freben]

For context, I see this comment in the discovery processor:

backstage/plugins/catalog-backend-module-gitlab/src/GitLabDiscoveryProcessor.ts

Line 153 in 45681c2

// https://gitlab.com/groupA/teams/teamA/subgroupA/repoA/-/blob/branch/filepath

The format expected by the GitLabUrlReader:
https://gitlab.com/groupA/teams/teamA/subgroupA/repoA/-/blob/branch/filepath

This unfortunately will trigger another API call in getGitLabFileFetchUrl to get the project ID.
The alternative is using the buildRawUrl function, which does not support subgroups, so providing a raw
URL here won't work either.

[freben]

This is how the code then fetches the project ID based on the URL where the file is stored:

backstage/packages/integration/src/gitlab/core.ts

Line 103 in 45681c2

export async function getProjectId(

This is something that anybody can do at a later time, for example a processor that fills in these IDs in annotations in a pre-processing step. Unfortunately that would lead to more API hammering of your GitLab instance - but the calls could be heavily cached, so maybe that actually can be sidestepped to a large degree.

To be clear, the UrlReader can't have this type of knowledge, since it's way more low level and should not modify data.

[jwnmulder]

A solution that currently works very well with the backstage-plugin-gitlab plugin is using 'gitlab.com/project-slug' instead of 'gitlab.com/project-id'.

After adding the AnnotateScmSlugEntityProcessor to your catalog configuration like this

  builder.addProcessor(
    AnnotateScmSlugEntityProcessor.fromConfig(env.config),
  );

the project-slug annotation is automatically added for all entities originating from a GitLab repository.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.