[Bug][github_graphql] Extract Jobs task fails to extract any collected jobs

[benduncan-tote]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

github_graphql Extract Jobs task fails to create records based on jobs collected in the Collect Job Runs task.

What do you expect to happen

Extract Jobs should correctly marshal the raw data and map it to the the _tool_github_jobs table

How to reproduce

For the Collect Job Runs task, the raw data is a check run.
https://github.com/apache/incubator-devlake/blob/0ecfbb0304fba4ac3084eaced883592760c64cd1/backend/plugins/github_graphql/tasks/job_collector.go#L186-L200

However, the Extract Jobs task expects the raw data to be a check suite.
https://github.com/apache/incubator-devlake/blob/0ecfbb0304fba4ac3084eaced883592760c64cd1/backend/plugins/github_graphql/tasks/job_extractor.go#L54-L91

This results in checkSuite.CheckSuite.CheckRuns.Nodes always being nil, causing no jobs to be extracted.

Anything else

No response

Version

v1.0.2-beta6

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[dosubot]

Hello @benduncan-tote! I'm Dosu, a bot here to help you tackle bugs, answer questions, and support your contributions. I'm here to assist while you wait for a human maintainer. Let's get started!

The issue you're encountering is due to a mismatch between the data format expected by the Extract Jobs task and the actual data collected by the Collect Job Runs task. The Collect Job Runs task collects data as check runs, while the Extract Jobs task expects this data to be in the form of check suites. This discrepancy results in checkSuite.CheckSuite.CheckRuns.Nodes being nil, preventing the extraction of job records.

To resolve this issue, you need to modify the Extract Jobs task to handle check run data instead of check suite data. This involves updating the extraction logic to correctly process the structure of check runs collected by the Collect Job Runs task. You may need to adjust the data mapping and transformation logic to ensure that the check run data is correctly marshaled and mapped to the _tool_github_jobs table.

Review the relevant sections of the code in job_extractor.go and job_collector.go to understand how the data is currently being processed and make the necessary adjustments to align the data formats between the two tasks.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[kostas-petrakis]

This looks affecting the existing beta versions (we are currently running v1.0.2-beta7 @klesh is there a possibility to release a new version? As currently our jobs are not extracted.

SELECT COUNT(1) FROM devlake._raw_github_graphql_jobs WHERE params  = "{\"ConnectionId\":1,\"Name\":\"***\/****\"}";

Count: 4571

SELECT COUNT(1) FROM devlake._tool_github_jobs WHERE _raw_data_params = "{\"ConnectionId\":1,\"Name\":\"***\/****\"}";

Count: 0