Skip to content

fix(jira): update epic collector to use new API endpoint and include #8547

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
klesh merged 7 commits into from
Aug 27, 2025
Merged

fix(jira): update epic collector to use new API endpoint and include #8547

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
c53f598
fix(jira): update epic collector to use new API endpoint and include …
GoSimplicity Aug 21, 2025
cda87db
Merge branch 'main' into main
GoSimplicity Aug 25, 2025
55fe451
fix(jira): enhance epic collector to dynamically select API endpoint …
GoSimplicity Aug 25, 2025
59dc06e
fix(jira): update epic collector to use correct API endpoint for JIRA…
GoSimplicity Aug 25, 2025
ecb4feb
Merge branch 'main' into main
GoSimplicity Aug 25, 2025
c3a54ae
fix(jira): refactor epic collector to streamline API endpoint selecti…
GoSimplicity Aug 25, 2025
74141bc
Merge branch 'main' into main
GoSimplicity Aug 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 96 additions & 6 deletions backend/plugins/jira/tasks/epic_collector.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,17 +82,42 @@ func CollectEpics(taskCtx plugin.SubTaskContext) errors.Error {
jql = buildJQL(*apiCollector.GetSince(), loc)
}

err = apiCollector.InitCollector(api.ApiCollectorArgs{
// Choose API endpoint based on JIRA deployment type
if data.JiraServerInfo.DeploymentType == models.DeploymentServer {
logger.Info("Using api/2/search for JIRA Server")
err = setupApiV2Collector(apiCollector, data, epicIterator, jql)
} else {
logger.Info("Using api/3/search/jql for JIRA Cloud")
err = setupApiV3Collector(apiCollector, data, epicIterator, jql)
}
if err != nil {
return err
}
return apiCollector.Execute()
}

// JIRA Server API v2 collector
func setupApiV2Collector(apiCollector *api.StatefulApiCollector, data *JiraTaskData, epicIterator api.Iterator, jql string) errors.Error {
return apiCollector.InitCollector(api.ApiCollectorArgs{
ApiClient: data.ApiClient,
PageSize: 100,
Incremental: false,
UrlTemplate: "api/2/search",
Query: func(reqData *api.RequestData) (url.Values, errors.Error) {
query := url.Values{}
epicKeys := []string{}
for _, e := range reqData.Input.([]interface{}) {
epicKeys = append(epicKeys, *e.(*string))

input, ok := reqData.Input.([]interface{})
if !ok {
return nil, errors.Default.New("invalid input type, expected []interface{}")
}

for _, e := range input {
if epicKey, ok := e.(*string); ok && epicKey != nil {
epicKeys = append(epicKeys, *epicKey)
}
}

localJQL := fmt.Sprintf("issue in (%s) and %s", strings.Join(epicKeys, ","), jql)
query.Set("jql", localJQL)
query.Set("startAt", fmt.Sprintf("%v", reqData.Pager.Skip))
Copy link
Copy Markdown
Collaborator

@narrowizard narrowizard Aug 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the information provided in the Atlassian developer changelog, the new api/3/search/jql endpoint has changed the way data is paginated. Specifically, random page access has been replaced with a continuation token API. This means that you won't be able to fetch multiple pages simultaneously using parallel threads. The startAt parameter has been replaced with nextPageToken.

Given these changes, simply updating the API endpoint without adjusting the pagination logic will result in incomplete data retrieval. The existing code that relies on the startAt parameter for pagination will no longer work as expected. It's crucial to implement the new nextPageToken mechanism to handle pagination properly and ensure that all data can be fetched correctly.

We’ll replace random page access with a continuation token API. This means you won’t be able to get multiple pages at the same time with parallel threads. startAt parameter will be replaced with nextPageToken.

Copy link
Copy Markdown
Contributor Author

@GoSimplicity GoSimplicity Aug 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent catch on the pagination changes! You're absolutely correct that api/3/search/jql uses nextPageToken instead of startAt. I'll update the implementation to:
1.Use different pagination strategies based on the API version
2.Set concurrency to 1 for api/3 (sequential pagination only)
3.Handle the nextPageToken properly in the response parser
I'll revise the PR to address these issues. Would you prefer separate collector implementations for each API version, or a single collector with conditional logic?

Copy link
Copy Markdown
Contributor

@klesh klesh Aug 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There’s already a mechanism for handling this kind of API. Here’s how you can enable it:

  1. Specify GetNextPageCustomData in ApiCollectorArgs to extract the nextPageToken.

    • This token will be stored in the RequestData.CustomData field.
    • Note: this also forces the collector to run in sequential mode (no parallel fetching).

  2. In the Query function, read the CustomData and plug it into the query string.

Here’s an example:
https://github.com/apache/incubator-devlake/blob/52433a8bc098eba668a02d815d618409aa375b93/backend/plugins/bitbucket_server/tasks/pr_collector.go#L47-L48

Expand All @@ -117,13 +142,78 @@ func CollectEpics(taskCtx plugin.SubTaskContext) errors.Error {
}
return data.Issues, nil
},
// Jira Server returns 400 if the epic is not found
AfterResponse: ignoreHTTPStatus400,
})
}

// JIRA Cloud API v3 collector
func setupApiV3Collector(apiCollector *api.StatefulApiCollector, data *JiraTaskData, epicIterator api.Iterator, jql string) errors.Error {
return apiCollector.InitCollector(api.ApiCollectorArgs{
ApiClient: data.ApiClient,
PageSize: 100,
Incremental: false,
UrlTemplate: "api/3/search/jql",
GetNextPageCustomData: getNextPageCustomDataForV3,
Query: func(reqData *api.RequestData) (url.Values, errors.Error) {
query := url.Values{}
epicKeys := []string{}
for _, e := range reqData.Input.([]interface{}) {
epicKeys = append(epicKeys, *e.(*string))
}
localJQL := fmt.Sprintf("issue in (%s) and %s", strings.Join(epicKeys, ","), jql)
query.Set("jql", localJQL)
query.Set("maxResults", fmt.Sprintf("%v", reqData.Pager.Size))
query.Set("expand", "changelog")
query.Set("fields", "*all")

if reqData.CustomData != nil {
query.Set("nextPageToken", reqData.CustomData.(string))
}

return query, nil
},
Input: epicIterator,
ResponseParser: func(res *http.Response) ([]json.RawMessage, errors.Error) {
var data struct {
Issues []json.RawMessage `json:"issues"`
}
blob, err := io.ReadAll(res.Body)
if err != nil {
return nil, errors.Convert(err)
}
err = json.Unmarshal(blob, &data)
if err != nil {
return nil, errors.Convert(err)
}
return data.Issues, nil
},
AfterResponse: ignoreHTTPStatus400,
})
}

// Get next page token for API v3
func getNextPageCustomDataForV3(_ *api.RequestData, prevPageResponse *http.Response) (interface{}, errors.Error) {
var response struct {
NextPageToken string `json:"nextPageToken"`
}

blob, err := io.ReadAll(prevPageResponse.Body)
if err != nil {
return err
return nil, errors.Convert(err)
}
return apiCollector.Execute()

prevPageResponse.Body = io.NopCloser(strings.NewReader(string(blob)))

err = json.Unmarshal(blob, &response)
if err != nil {
return nil, errors.Convert(err)
}

if response.NextPageToken == "" {
return nil, api.ErrFinishCollect
}

return response.NextPageToken, nil
}

func GetEpicKeysIterator(db dal.Dal, data *JiraTaskData, batchSize int) (api.Iterator, errors.Error) {
Expand Down
Loading