[Bug][Github] Github deployments without Github Actions

[carocad]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

The devlake documentation says that Github deployment are read in order to compute DORA metrics. However, in several of my (private) projects, I have Github deployments that are created without using Github Actions. In those cases, devlake doesn't detect the Deployments as (it seems to) assume that Deployments can only come from Github Actions as seen here.

What do you expect to happen

devlake should take the Deployment information from the Github Deployment API not from the Github Actions API; see docs. As per the previous docs it is possible to create Github deployments without using Github actions.

How to reproduce

• setup a github repository
• setup a ci/cd system outside of Github that creates Github Deployments when doing a Deployment
• setup devlake to connect to Github and fetch ci/cd information from there
• Deployment data will be missing even though Deployments are clearly visible on the Github UI

Anything else

See steps above. It happens every time even with the latest version

Version

v1.0.0-beta6

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[klesh]

I think we do collect Github Deployment as well https://github.com/apache/incubator-devlake/blob/470669d603f792927d1313194a408597a1c145fd/backend/plugins/github_graphql/tasks/deployment_collector.go#L42

Select the CICD entities in the configuration and it should be good to go, just ignore the Github Action if you don't need them.

[carocad]

Maybe I misunderstood the code (I haven't taken a deep look) but it looks like that is only used by the graphql implementation but in our case we are using the "plain api".

just ignore the Github Action if you don't need them.

We tried this, we have 2 projects :

• one with Github deployments but no Github Action and
• another one with Github deployments through Github Actions

In the first case we don't get any deployment information shown in devlake/grafana dashboards but on the second case we do get the information which is what pointed me to the distinction mentioned on the bug description.

Please let me know if I can provide you some debug logs or similar to further clarify this :)
PS: we are using Github Enterprise edition not github.com

[klesh]

That is weird, the subtasks for collecting and converting Github Deployment are enabled by default.
Can you share the configuration which you can find by following this screenshot

[carocad]

That is weird, the subtasks for collecting and converting Github Deployment are enabled by default. Can you share the configuration which you can find by following this screenshot

Sure thing, here you go.

ghe+actions.json
ghe+jenkins.json

The first file is the one where deployments are made with github actions; whereas the second one is where the deployments are made outside of it. On the json files I can see that "Collect Workflow Runs" is a subtasks but there is no "Collect Deployments" or similar.

[klesh]

I see, did you disable the github graphql feature on the connection detail?

[klesh]

Try enable it and see if the problem got fixed.

[carocad]

Try enable it and see if the problem got fixed.

just tried it but unfortunately it crashes. See logs below

time="2024-05-14 08:46:28" level=info msg=" [pipeline service] [pipeline #87] [task #618] start executing task: 618"
time="2024-05-14 08:46:28" level=info msg=" [pipeline service] [pipeline #87] [task #618] start plugin"
time="2024-05-14 08:46:28" level=info msg=" [pipeline service] [pipeline #87] [task #618] [api async client] creating scheduler for api \"https://github.boschdevcloud.com/api/v3/\", number of workers: 13, 9500 reqs / 1h0m0s (interval: 378.947368ms)"
time="2024-05-14 08:46:28" level=error msg=" [pipeline service] [pipeline #87] [task #618] run task failed with panic\n\tcaused by: run task failed with panic (github.com/apache/incubator-devlake/helpers/pluginhelper/api.CreateAsyncGraphqlClient:71)\n\tWraps: (2) non-200 OK status code: 404 Not Found body: \"{\\\"message\\\":\\\"Not Found\\\",\\\"documentation_url\\\":\\\"https://docs.github.com/enterprise-server@3.11/rest\\\"}\"\n\tWraps: (3) non-200 OK status code: 404 Not Found body: \"{\\\"message\\\":\\\"Not Found\\\",\\\"documentation_url\\\":\\\"https://docs.github.com/enterprise-server@3.11/rest\\\"}\"\n\tError types: (1) *hintdetail.withDetail (2) *hintdetail.withDetail (3) *errors.errorString"

[klesh]

Seems like you are using the enterprise version, is it Cloud or On-premise?

[carocad]

Seems like you are using the enterprise version, is it Cloud or On-premise?

GitHub Enterprise Server 3.11.9 (OnPremise)

[klesh]

It sounds like there might be some specific behaviors with GitHub Enterprise Server 3.11.9 (On-Premise). Unfortunately, since we don't currently have access to this version for testing, it limits our ability to directly replicate the issue.

Here are a few ways we can move forward:

Community Resources: Have you checked the GitHub Enterprise Server documentation or community forums for known quirks or workarounds related to your specific version? There might be existing solutions or insights from other users.
Consider Upgrading (Optional): If feasible, upgrading to a newer version of GitHub Enterprise Server might resolve the issue and provide access to the latest features and bug fixes.

[carocad]

Hello @klesh ,
It seems that the rate limit behavior is deactivated by default on Github Enterprise Server. See docs: https://docs.github.com/en/enterprise-server@3.11/graphql/overview/rate-limits-and-node-limits-for-the-graphql-api#primary-rate-limit

Rate limits are disabled by default for GitHub Enterprise Server.

Would it be possible for devlake to handle that case with a default value or another method which avoid a panic? I can also contact the administrators on our side and ask if they can set one but since this is the default behavior from Github Server I would assume that this wouldn't be an isolated case.

[klesh]

@carocad Sure, you may find the "Custom Rate Limit" on the connection page.

[carocad]

@carocad Sure, you may find the "Custom Rate Limit" on the connection page.

Unfortunately that didn't work either. See logs below

time="2024-05-21 07:48:30" level=info msg=" [pipeline service] [pipeline #145] [task #971] start executing task: 971"
time="2024-05-21 07:48:30" level=info msg=" [pipeline service] [pipeline #145] [task #971] start plugin"
time="2024-05-21 07:48:30" level=info msg=" [pipeline service] [pipeline #145] [task #971] [api async client] creating scheduler for api \"https://github.boschdevcloud.com/api/v3/\", number of workers: 6, 4500 reqs / 1h0m0s (interval: 800ms)"
time="2024-05-21 07:48:31" level=error msg=" [pipeline service] [pipeline #145] [task #971] run task failed with panic\n\tcaused by: run task failed with panic (github.com/apache/incubator-devlake/helpers/pluginhelper/api.CreateAsyncGraphqlClient:71)\n\tWraps: (2) non-200 OK status code: 404 Not Found body: \"{\\\"message\\\":\\\"Not Found\\\",\\\"documentation_url\\\":\\\"https://docs.github.com/enterprise-server@3.11/rest\\\"}\"\n\tWraps: (3) non-200 OK status code: 404 Not Found body: \"{\\\"message\\\":\\\"Not Found\\\",\\\"documentation_url\\\":\\\"https://docs.github.com/enterprise-server@3.11/rest\\\"}\"\n\tError types: (1) *hintdetail.withDetail (2) *hintdetail.withDetail (3) *errors.errorString"

I can only assume from this that DevLake would check the rate limit endpoint regardless of the custom rate limit and only after it receives a response it would then overwrite it. If you agree that this is a bug, I would happily try to create a PR to fix it :)

[carocad]

... I can also contact the administrators on our side and ask if they can set one but since this is the default behavior from Github Server I would assume that this wouldn't be an isolated case.

I just got an answer from the administrators on my site. The GraphQL endpoints on our side already have a rate limit so this shouldn't be the case. I also tried querying it myself and got an response without any issues. I submitted from the github.com docs (see ref)

query {
  viewer {
    login
  }
  rateLimit {
    limit
    remaining
    used
    resetAt
  }
}

response

{
    "data": {
        "viewer": {
            "login": "<redacted>"
        },
        "rateLimit": {
            "limit": 5000,
            "remaining": 4999,
            "used": 1,
            "resetAt": "2024-05-21T10:17:01Z"
        }
    }
}

I don't know which query does dev-lake uses but I can only guess that one of the fetched parameters was introduced on a version of github newer than 3.11.9. In any case, my proposal above still stands.
If you point me in the right direction I can definitely give it a try :)