fix(jira): stop the changelog convertor hiding unconverted items forever

[vbhanuchander-lang]

Closes #8834

The mechanism is the incremental filter, not the LEFT JOIN

The thread converged on the LEFT JOIN + WHERE board_id anti-pattern, and @klesh reasonably rejected the proposed fix because moving board_id into the ON clause makes every board task convert the whole connection. I think the actual defect is somewhere else, and the fix costs nothing in scope or performance.

The convertor's incremental filter is:

dal.Where("_tool_jira_issue_changelog_items.created_at >= ? ", since)

created_at comes from common.NoPKModel and is stamped when the row is first inserted. It never moves again. So a changelog item collected during one window but not converted in that window can never be selected by any later incremental run — the column it is filtered on is permanently in the past. Nothing errors. The rows just stay in the tool layer.

That fits every symptom in the report:

Reported	Explained by
Partial, ~79% converted / ~21% not, same sync run	window boundary, not board membership
Persists across runs	created_at cannot move, so the item is permanently excluded
No errors, no warnings	nothing failed — the rows were never selected
4%–99% across projects	depends on how much of each project's history predates the window

It also explains why the "not the incremental filter" check came out clean: _devlake_collector_latest_state is the collector's state. The convertor keeps its own subtask state, and that is what gates this query.

The fix

-dal.Where("_tool_jira_issue_changelog_items.created_at >= ? ", since)
+dal.Where("_tool_jira_issue_changelog_items.updated_at >= ? ", since)

updated_at is refreshed by the extractor's upsert (CreateOrUpdate → OnConflict{UpdateAll: true}), so a re-collected item is reconsidered and a re-collection actually repairs the gap.

Consistency check across the code base — of 36 convertors, this was the only one filtering on created_at:

$ grep -rh "dal.Where(\"" --include="*convertor*.go" plugins/ | grep -E "(created|updated)_at >= "
  35 updated_at
   1 created_at   <- plugins/jira/tasks/issue_changelog_convertor.go

Crucially, this does not widen the board filter. No board task converts anything outside its own board, so the cost @klesh objected to does not arise.

Making the board filter non-silent

The board filter is deliberate and I have left it alone. But the issue also asks that the convertor not skip records silently, and that part is fair independently of the bug above.

After a successful conversion the subtask now counts collected changelog items whose issue is not on this board, and logs the number with the reason. One aggregate query per board task, negligible next to the conversion. It turns "21% of my changelogs are missing" into a logged figure with an explanation. A failure in the diagnostic is logged, never propagated — it cannot turn a good run into a failed one.

@ciaramulligan — for overlapping boards this means the shortfall becomes visible and attributable rather than mysterious, and with the filter fix a re-collection now repairs previously stranded items.

Tests

TestIssueChangelogBoardScopeDataFlow (e2e, ran against MySQL 8.4) drives the convertor with a changelog for an issue on no board and asserts: the out-of-scope item is not converted, nothing is attached to jira:JiraIssue:2:0, and in-scope changelogs still convert — so the exclusion is scoping, not a regression that dropped everything. The diagnostic is observably firing in that run:

level=warning msg="1 collected changelog item(s) are not associated with board 8 and were not
converted; they belong to issues outside this board's scope. ..."

TestIncrementalFilterUsesUpdatedAtNotCreatedAt guards the column choice. Reverting it is a one-token change that silently restores permanent data loss and nothing else in the suite would catch it.

Full plugin, with a database attached:

ok  github.com/apache/incubator-devlake/plugins/jira/api               0.221s
ok  github.com/apache/incubator-devlake/plugins/jira/e2e              11.106s
ok  github.com/apache/incubator-devlake/plugins/jira/tasks            0.254s
ok  github.com/apache/incubator-devlake/plugins/jira/tasks/apiv2models 0.418s

What I could not verify

I have no access to the reporter's Jira instance, so I have not reproduced the original 21% shortfall end to end. The mechanism above is established from the code, the model definitions and the upsert behaviour, and it is consistent with every reported symptom — but the confirmation that would close it beyond doubt is @ciaramulligan running a sync on this branch and comparing tool-layer and domain-layer counts again.

If the shortfall persists after this, the remaining suspect is genuine board non-membership, and the new warning will say exactly how many items that accounts for — which turns the next round of diagnosis into reading a log line instead of writing SQL.

[klesh]

LGTM
Thanks for your contribution.

[ciaramulligan]

Thanks @vbhanuchander-lang — We've been working around the bug by querying the tool tables directly instead of issue_changelogs, and the pattern you identified matches what we were seeing. Smaller boards were barely affected but some of our larger multi-project boards had significant data loss.

I checked the missing items and they consistently have created_at stuck at first-insert time while updated_at was refreshed by re-extraction — like you describe.

I'll validate the fix end-to-end once we upgrade our Devlake version and report back with the results. Thank you!

[vbhanuchander-lang]

Thanks for confirming it against real data — created_at stuck at first-insert while
updated_at moves with re-extraction is exactly the signature, and it's useful to know it
scaled with board size for you.

One thing worth planning for before you validate: upgrading alone will not backfill the
rows you have already lost. The new filter is still only applied on incremental runs:

if stateManager.IsIncremental() {
    since := stateManager.GetSince()
    if since != nil {
        clauses = append(clauses, dal.Where("..._changelog_items.updated_at >= ? ", since))
    }
}

Your historical gaps have an updated_at older than the since of the next incremental
run, so they stay outside the window. Two ways to recover them, either is fine:

• Run a full-refresh sync on the affected boards. IsIncremental() is false, no time
  filter is added, and every collected changelog item in board scope is converted.
• Or re-collect the issues first — the extractor upserts with OnConflict{UpdateAll: true},
  which refreshes updated_at, after which a normal incremental run reconsiders them.

After that, if anything still looks missing, the PR also adds a diagnostic that should tell
you whether it is scope rather than a bug. It logs, once per board:

N collected changelog item(s) are not associated with board <id> and were not converted; they belong to issues outside this board's scope.

Previously those items were silently dropped, so "missing" and "out of scope" were
indistinguishable in the logs. If that warning accounts for your remaining delta, the data
is being excluded by board scope by design and you would need a board covering those issues.

Please do report back — if the numbers don't line up after a full refresh I would like to
know, since that would point at something beyond this fix.