[Bug][CircleCI Plugin] CircleCI pipelines collected from before time range

[Nickcw6]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

When collecting CircleCI pipelines, the time range specified in the sync policy has no effect on the data collected - pipelines are collected from before the specified date.

E.g. Sync policy settings set to collect from 1st June 2024:

Excerpt of data JSON blob from top row - has created_date and updated_date of 1st Feb 2024 (ie. 180 days ago from todays date - 2024-07-30):

{
    "id" : "eae60b4c-7dcc-4293-8b00-45f18a494881",
    "updated_at" : "2024-02-01T14:11:31.988Z",
    "created_at" : "2024-02-01T14:11:31.988Z",
    ...
    "trigger" : {
      "received_at" : "2024-02-01T14:11:31.481Z",
      "type" : "webhook",
    ...
    },
    ...
  }

What do you expect to happen

No CircleCI pipelines, workflows or jobs are collected from before the time range start point.

How to reproduce

1. Set the time frame to a date before any CircleCI data retention period ends (e.g. if retention period is 90 days, set this to 30 days - see below).
2. Run the DevLake pipeline
3. Sort the _raw_circleci_api_pipelines table by the created_at JSON property of the data column:

SELECT *
FROM _raw_circleci_api_pipelines
ORDER BY STR_TO_DATE(JSON_UNQUOTE(JSON_EXTRACT(CONVERT(data USING utf8mb4), '$.created_at')), '%Y-%m-%dT%H:%i:%s.%fZ') ASC;

4. Compare the created_at property to that set in the sync policy time range.

Anything else

This is in part due to the recent pagination fix on the plugin (#7770) - the pagination works but as the CircleCI API does not offer any date range pagination controls, the collector now loops through the pages until next_page_token is null, which is whenever the data retention limit is hit for the account (e.g. for me it is 180 days, but could be less/more, see here).

When subsequently attempting to collect the relevant workflows & jobs for the pipeline, this will return a 404 and error the DevLake pipeline for any data points that fall outside of the data retention range in a race condition vs. CircleCI cleaning up build data:

subtask collectWorkflows ended unexpectedly Wraps: (2) Retry exceeded 3 times calling /v2/pipeline/6b7c4513-56bd-4e0c-ad72-d562df7513b1/workflow. The last error was: Http DoAsync error calling [method:GET path:/v2/pipeline/6b7c4513-56bd-4e0c-ad72-d562df7513b1/workflow query:map[]]. Response: {:message "Pipeline not found"} (404) Error types: (1) *hintdetail.withDetail (2) *errors.errorString

There needs to be an additional check that the created_at property of the returned pipelines is not before the specified time range starting point.

Version

44c3ecb

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[dosubot]

To address the issue of CircleCI pipelines being collected from before the specified time range, you can implement a check in the CollectPipelines function to filter out pipelines based on their created_at property. This will ensure that only pipelines created within the specified time range are collected.

Here's a step-by-step guide on how to implement this:

1. Modify the CollectPipelines function:

   • Add a check to filter pipelines based on the created_at property before storing them in the database.

2. Update the CollectPipelines function:

   • Add a parameter to pass the start date of the time range.
   • Filter the pipelines based on the created_at property.

Here is an example of how you can modify the CollectPipelines function:

func CollectPipelines(taskCtx plugin.SubTaskContext) errors.Error {
    rawDataSubTaskArgs, data := CreateRawDataSubTaskArgs(taskCtx, RAW_PIPELINE_TABLE)
    logger := taskCtx.GetLogger()
    logger.Info("collect pipelines")

    // Define the start date for the time range filter
    startDate := data.Options.TimeAfter

    collector, err := api.NewApiCollector(api.ApiCollectorArgs{
        RawDataSubTaskArgs: *rawDataSubTaskArgs,
        ApiClient:          data.ApiClient,
        UrlTemplate:        "/v2/project/{{ .Params.ProjectSlug }}/pipeline",
        PageSize:           int(data.Options.PageSize),
        GetNextPageCustomData: func(prevReqData *api.RequestData, prevPageResponse *http.Response) (interface{}, errors.Error) {
            res := CircleciPageTokenResp[any]{}
            err := api.UnmarshalResponse(prevPageResponse, &res)
            if err != nil {
                return nil, err
            }
            if res.NextPageToken == "" {
                return nil, api.ErrFinishCollect
            }
            return res.NextPageToken, nil
        },
        Query: func(reqData *api.RequestData) (url.Values, errors.Error) {
            query := url.Values{}
            if pageToken, ok := reqData.CustomData.(string); ok && pageToken != "" {
                query.Set("page_token", reqData.CustomData.(string))
            }
            return query, nil
        },
        ResponseParser: func(res *http.Response) ([]json.RawMessage, errors.Error) {
            data := CircleciPageTokenResp[[]json.RawMessage]{}
            err := api.UnmarshalResponse(res, &data)
            if err != nil {
                return nil, err
            }

            // Filter pipelines based on the created_at property
            filteredItems := []json.RawMessage{}
            for _, item := range data.Items {
                var pipeline map[string]interface{}
                if err := json.Unmarshal(item, &pipeline); err != nil {
                    return nil, errors.Convert(err)
                }
                createdAtStr, ok := pipeline["created_at"].(string)
                if !ok {
                    continue
                }
                createdAt, err := time.Parse(time.RFC3339, createdAtStr)
                if err != nil {
                    return nil, errors.Convert(err)
                }
                if createdAt.After(startDate) {
                    filteredItems = append(filteredItems, item)
                }
            }
            return filteredItems, nil
        },
    })
    if err != nil {
        logger.Error(err, "collect pipelines error")
        return err
    }
    return collector.Execute()
}

In this example:

• The startDate is retrieved from the data.Options.TimeAfter parameter.
• The ResponseParser function filters the pipelines based on the created_at property, ensuring that only pipelines created after the startDate are collected.

This approach ensures that the CircleCI pipelines collected are within the specified time range in the sync policy, preventing the collection of pipelines created before the specified date.

For more details on configuring the CircleCI plugin and setting the sync policy, you can refer to the CircleCI configuration documentation ^[1][2].

To continue talking to Dosu, mention @dosu.

[klesh]

Apache DevLake doesn't support Data Retention.
If you collect data in 180 days and then reduce the Time Range to 90 days, all collected data stay in the database, it is by design.

[Nickcw6]

@klesh To clarify - in the example above the only data collection ran was configured from 2024-06-01 - data preceding this should not have been collected. The data retention is from CircleCI and affects the results returned from their API.

[klesh]

@Nickcw6 Thanks for the clarification. It could be a problem then. Are you running the devlake from the source code? If so, would you like to work on the problem ?

[Nickcw6]

@klesh Yep - going to give it a go

[Nickcw6]

Hey @klesh, I believe i've fixed the issue in #7820 - however I have a question on your comment here:

If you collect data in 180 days and then reduce the Time Range to 90 days, all collected data stay in the database, it is by design.

Is this also true for plugins where only Full Refresh is supported (ie this CircleCI plugin as per here)?

As after fixing this bug I notice that for the circleci tables, if I run a collection from 2024-07-01 (for example), then run another from 2024-07-25 (not in full refresh mode), all the previous data is removed and the earliest data point will be from 2024-07-25 onwards.

E.g. initial 2024-07-01 run (779 rows in _raw_circleci_pipelines), ordered by created_at property in data column:

Excerpt from top row:

{
    "id": "82a54e28-cdbb-4654-a0c4-e043b7ddf920",
    "updated_at": "2024-07-01T05:37:12.933Z",
  	"created_at": "2024-07-01T05:37:12.933Z",
....

}

After rerunning from 2024-07-25 - only 141 rows in _raw_circleci_pipelines - top record - NOT in full refresh mode:

{
    "id": "965dae3d-09b7-42ad-a582-8029a2071693",
    "updated_at": "2024-07-25T07:06:50.288Z",
    "created_at": "2024-07-25T07:06:50.288Z"
    ....
}

Is this the expected behaviour?

[klesh]

@Nickcw6 Good catch. Yes, it is expected. I failed to mention that I was talking about subtasks that support Incremental Collection.

[Nickcw6]

Okay cool - thanks @klesh.

I'm going to investigate how feasible implementing incremental sync for the plugin is. As in its current state with the data retention policy we'd only be to able to display max 6 months worth of data - not the end of the world - but additionally due to circleci's api the data collection for that length of time is pretty slow - it took >16 hours for one repo (an active mono-repo, but still a long time).

[klesh]

Nice, feel free to ask me anything if you run into problems.
You may reach out to me on Slack(link can be found on the homepage of the project), and we may create a dedicated channel to discuss the development if you will.