[Bug][gitlab] duration_sec has wrong value

[antoniomuso]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

The duration_sec attribute in the cicd_pipelines table Gitlab has the wrong value. His value is not the value taken by the API, but it is evaluated here https://github.com/apache/incubator-devlake/blob/d5276df04c02e7df0587b4a7616ddfe01625e5c0/backend/plugins/gitlab/tasks/pipeline_extractor.go#L86 .

What do you expect to happen

I expect to have the value returned by the API https://docs.gitlab.com/ee/api/pipelines.html#get-a-single-pipeline

How to reproduce

Query the cicd_pipelines you see that the value returned by the query is different from the one returned by the GitLab API

Anything else

No response

Version

v0.18.0

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[klesh]

thanks for reporting the bug.

[abeizn]

@antoniomuso Sorry, I'm afraid we can't implement it this way, because this must be fully collected in pipeline detail, which will greatly slow down our code collection. So we made a trade-off and gave up getting the value from the pipeline detail API.

[antoniomuso]

Hi @abeizn, can you explain the tradeoff you made?

[klesh]

### Prioritize Page-Based Data Collection for Efficiency

Collecting pipelines individually through the /pipelines/:id API endpoint is significantly slower compared to retrieving them in pages using the /pipelines endpoint. Here's a breakdown illustrating the efficiency gains:

Scenario:

• Total pipelines: 1000
• Records per page (using /pipelines): 100

API Requests Required:

• Page-based collection (/pipelines): 10 requests
• Individual collection (/pipelines/:id): 1000 requests

Key Points:

• Page-based retrieval effectively reduces the number of API calls by a factor of 100 in this example, leading to substantial performance improvements.
• This approach is particularly advantageous when dealing with large datasets.
• Always prioritize page-based collection methods whenever possible to optimize data retrieval efficiency.

[abeizn]

@antoniomuso Hello, after discussion, we found that there were some problems with the previous code implementation logic, and we really needed to execute all pipeline details, so we fixed this bug and it will take effect in v0.20.0-beta6.