[Bug][Github_graphql] Job_collector not paginating correctly

[FlomoN]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

We have repositories with github actions pipelines that consistently have around 30+ Jobs to a run and we noticed we were missing many deployments (which was usually the last job of a run) to be reported in the DORA metrics.

What do you expect to happen

All deployments should be identified correctly, even for github action workflow runs with more than 20 jobs.

How to reproduce

Have multiple github actions workflow runs with more than 20 jobs representing a deployment

Anything else

By peeking at the raw tables in the database we could determine, that these runs that were not detected correctly only ever had exactly 20 jobs in the database instead of the full 30+. So something seems wrong with the pagination of the graphql query.

Version

v1.0.3-beta6

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[dosubot]

Hi @FlomoN! I'm Dosu and I’m helping the incubator-devlake team.

You're correct—there's a pagination issue in the GitHub GraphQL job_collector. The code sets a page size of 20 when fetching jobs, and while it is designed to iterate over pages using the hasNextPage and endCursor from the GraphQL response, it appears that only the first page is being processed, which matches your observation of only 20 jobs per run being stored in the database. This results in missing jobs for workflow runs with more than 20 jobs, impacting DORA metrics and deployment detection. You can review and update the pagination logic in job_collector.go to ensure it continues fetching jobs until hasNextPage is false and all jobs are collected.

If you're planning a PR, focus on verifying that the collector loops through all pages for each workflow run, using the endCursor to fetch subsequent pages. The relevant pagination handling is in the getPageInfo and CollectJobs functions. There are no configuration options to change the page size or pagination behavior in the current release, so a code fix is needed. Recent PRs have improved data correctness but haven't addressed this specific bug yet. Let me know if you need pointers to specific code sections or want to discuss implementation details!

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[klesh]

Thanks for putting up the issue and working on it.
Looking forward to your PR.
Feel free to cue me or reach out to me on Slack if you need help

Didn't notice the PR is ready. 😂 Good work.