[Feature][AzureDevOps/GitHub] Support extracting Azure Pipelines metrics from GitHub Repositories

[Feggah]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Use case

I have GitHub Repositories (Data Scopes of GitHub Plugin) and I use Azure Pipelines as a CI/CD tool. Azure Pipelines is part of AzureDevOps.

I want to be able to select GitHub Repositories and configure DevLake to extract DORA metrics from Azure Pipelines + GitHub Repo. But, today, I can only use GitHub Actions with GitHub Repositories.

Description

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Startrekzky]

Got it. May I know why you use this combination of GitHub repo and Azure Pipeline? Are you maintaining an open-source project? I want to know if it is a common use case.

[Feggah]

We (as a company) decided to use GitHub as the tool to version our repositories. As we grew bigger, each team/tribe decided to use a different CI/CD tool to manage their applications.

This means that we have all possible combinations of tools, for example:

• Github Repos + GitHub Actions (supported by DevLake)
• GitHub Repos + Azure Pipelines (not supported, that's my feature request)
• GitHub Repos + Azure Releases
• GitHub Repos + CircleCI
• etc...

I don't know if this is a common use case in another companies (I imagine it is), but here we have 2k+ applications with this usecase.

[Startrekzky]

Thank you, @Feggah . This is pretty informative. I just found out we seem to have supported this feature in #5249. However, I didn't find any docs about how to configure it.

@CamilleTeruel Do you have any idea how we can use this feature?

[keon94]

@Feggah We do support Github-exclusive repos. They are prefixed with GitHub in the data-scopes seletion window. Example:

That points to https://github.com/keon94/cicd-playground which has azure pipelines configured.

[Feggah]

I cannot reproduce your test @keon94 . Is there any special configuration?

I have a AzureDevOps connection, but I can only see Azure Repositories. Do I need to select something special?

[keon94]

@Feggah There shouldn't be special config. Is this a private repo?
Try this for troubleshooting (which is what the backend code does). Use POSTMAN or curl using your Azure devops token (Basic Auth with Password only) to:

1. Call GET https://dev.azure.com/${YOUR_ORG_NAME}/${YOUR_PROJECT_NAME}/_apis/serviceendpoint/endpoints. Look in the returned JSON for the field type, and make sure you see GitHub, and the field id for it (corresponds to a github org).
2. Call GET https://dev.azure.com/${YOUR_ORG_NAME}/${YOUR_PROJECT_NAME}/_apis/sourceProviders/GitHub/repositories?serviceEndpointId=${ID_FROM_ABOVE} and see if your github repo is in there.

In my case I get this coming back in the payload:

[Feggah]

Is this a private repo?

Yes, they are private

---

Using POSTMAN it worked. I got a lot of repositories from AzureDevOps API. See below:

GET https://dev.azure.com/REDACTED/REDACTED/_apis/sourceProviders/GitHub/repositories?serviceEndpointId=REDACTED

But I also have a lot of ServiceConnections (or ServiceEndpoints) in my AzureDevOps Project. There are multiple GitHub connections. Maybe that's why it is not displaying in DevLake?

https://dev.azure.com/REDACTED/REDACTED/_apis/serviceendpoint/endpoints

Some thoughts:

• Maybe DevLake is selecting a ServiceConnection without the necessary permissions? What connection DevLake chooses when there are a lot of options?
• The GET request that returned 2000 repositories took 40.2 seconds. Maybe DevLake is timeouting the request?

[Startrekzky]

Hi @keon94 , I tested it in v0.18.0-alpha17 (around beta1 I guess?) and the results were:

1. When adding a data scope to an Azure connection, I can fetch the external Github repo
   

2. However, after I added it to the connection, it didn't show the repo's name.
   

3. When I tried to delete the scope, it was not deletable.
   

4. When I tried to add the connection to a project and collect the data of the external repo, it threw an error:

error patching the blueprint (400) Wraps: (2) Wraps: (3) make plan for blueprint failed (400) Wraps: (4) Invalid scope id (400)"

which seemed to have something to do with the scope id.

[keon94]

@Startrekzky Invalid scope ID is likely happening because UI is sending an empty scope to the backend. As far as the list of scopes being empty, I think this has been resolved in beta2, by #5708.

[Startrekzky]

@keon94 Got it. I'll re-test it in beta3 and let you know the results.

[Feggah]

@Startrekzky could you also check if you can see this GitHub repository when creating a Blueprint instead of adding directly to the Connection?

Because I can't see any GitHub repos from an AzureDevOps Connection when creating a blueprint. Maybe there is a bug in that flow

[Startrekzky]

Hi @Feggah , the v0.18.0's flow of adding repos has changed. I can only add data scope (external repos in this case) in the Data Connection page now.

And I just verified that in v0.18.0-beta3, when I tried to add repos to an Azure DevOps connection, it could list out the external GitHub repos(see the pic). And the data was successfully collected in a DevLake project. cc. @keon94

@Feggah Can you show me the screenshot of adding the external data scope in your v0.17.0-beta13? Perhaps it's a bug in that version. I'm not sure.

[Feggah]

@Startrekzky I upgraded to v0.18.0-beta3 and tried to add the external repository. The problem now is that the repositories are loading forever:

By looking at the logs of the DevLake API:

I think the problem is that my AzureDevOps project has a lot of Service Connections with different types. The fix probably would be skipping Service Connections which are not type == GitHub? Then it will only use valid Service Connections to find repositories

[Startrekzky]

Hi @Feggah, sorry that I missed the notifications. I'll forward it to @klesh to take a look.

[klesh]

@CamilleTeruel Can you take a look at this issue?

[CamilleTeruel]

fixed via #5950

[Startrekzky]

Hi @Feggah , this issue is fixed in https://github.com/apache/incubator-devlake/releases/tag/v0.18.0-beta8 . Please let me know if you still have this problem.