fix(#2562): Prevent transaction deduplication for imported transactions

[strazto]

Resolves #2562

Based on #2618 , with comments by @MatissJanis and others applied

[netlify]

✅ Deploy Preview for actualbudget ready!

Name	Link
🔨 Latest commit	3cbc56a
🔍 Latest deploy log	https://app.netlify.com/sites/actualbudget/deploys/664f915043ebef0008c70776
😎 Deploy Preview	https://deploy-preview-2770.demo.actualbudget.org
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[github-actions]

Bundle Stats — desktop-client

Hey there, this message comes from a GitHub action that helps you and reviewers to understand how these changes affect the size of this project's bundle.

As this PR is updated, I'll keep you updated on how the bundle size is impacted.

Total

Files count	Total bundle size	% Changed
9	4.71 MB	0%

Changeset

No files were changed

View detailed bundle breakdown

Added

No assets were added

Removed

No assets were removed

Bigger

No assets were bigger

Smaller

No assets were smaller

Unchanged

Asset	File Size	% Changed
static/js/indexeddb-main-thread-worker-e59fee74.js	13.5 kB	0%
static/js/resize-observer.js	18.37 kB	0%
static/js/BackgroundImage.js	122.29 kB	0%
static/js/usePreviewTransactions.js	790 B	0%
static/js/AppliedFilters.js	20.41 kB	0%
static/js/narrow.js	59.97 kB	0%
static/js/wide.js	262.45 kB	0%
static/js/ReportRouter.js	1.23 MB	0%
static/js/index.js	3 MB	0%

[github-actions]

Bundle Stats — loot-core

Hey there, this message comes from a GitHub action that helps you and reviewers to understand how these changes affect the size of this project's bundle.

As this PR is updated, I'll keep you updated on how the bundle size is impacted.

Total

Files count	Total bundle size	% Changed
1	1.2 MB → 1.2 MB (+239 B)	+0.02%

Changeset

File	Δ	Size
packages/loot-core/src/server/accounts/sync.ts	📈 +262 B (+1.16%)	22.04 kB → 22.29 kB

View detailed bundle breakdown

Added

No assets were added

Removed

No assets were removed

Bigger

Asset	File Size	% Changed
kcab.worker.js	1.2 MB → 1.2 MB (+239 B)	+0.02%

Smaller

No assets were smaller

Unchanged

No assets were unchanged

[MatissJanis]

I'm trying to wrap my head around the difference between #2618 and this PR, but I can't seem to really get it.

Would you mind creating a unit test case that passes in this PR, but would fail in #2618?

[pmoon00]

I'm trying to wrap my head around the difference between #2618 and this PR, but I can't seem to really get it.

Would you mind creating a unit test case that passes in this PR, but would fail in #2618?

From my interpretation @strazto is implying that if the imported transaction doesn't have an imported_id then it should include existing transactions that either have or don't have an imported_id.

The PR #2618 excludes all existing transactions with an imported_id in the fuzzy search.

I'm not super confident in what is technically correct here as you've mentioned some complex scenarios in comments in #2618, I'll let you be the judge @MatissJanis

I believe the test case that would pass in this PR but not in #2618 would be something like the below. I have tested this test fails in #2618 as the assertion for the number of transactions fails, receiving 2, when expecting 1.

  test(
    'given an imported tx with no imported_id, ' +
      'when an existing transaction that has an imported_id and matches amount and is within 7 days of imported tx,' +
      'then imported tx should reconcile with existing transaction from fuzzy match',
    async () => {
      const { id } = await prepareDatabase();

      let payees = await getAllPayees();
      expect(payees.length).toBe(0);

      const existingTx = {
        date: '2024-04-05',
        amount: -1239,
        imported_payee: 'Acme Inc.',
        payee_name: 'Acme Inc.',
        imported_id: 'b85cdd57-5a1c-4ca5-bd54-12e5b56fa02c',
        notes: 'TEST TRANSACTION',
        cleared: true,
      };

      // Add transaction to represent existing transaction with imoprted_id
      await reconcileTransactions(id, [existingTx]);

      payees = await getAllPayees();
      expect(payees.length).toBe(1);

      let transactions = await getAllTransactions();
      expect(transactions.length).toBe(1);

      // Import transaction similar to existing but with different date and no imported_id
      await reconcileTransactions(id, [
        {
          ...existingTx,
          date: '2024-04-06',
          imported_id: null,
        }
      ]);

      payees = await getAllPayees();
      expect(payees.length).toBe(1);

      transactions = await getAllTransactions();
      expect(transactions.length).toBe(1);

      expect(
        transactions.find(
          t => t.imported_id === 'b85cdd57-5a1c-4ca5-bd54-12e5b56fa02c',
        ).amount,
      ).toBe(-1239);
    },
  );

Edit: After making this comment I went and got this PR on my machine. The test needed a bit of tweaking to work but in essence asserted my thinking was correct. I did find an issue/side effect, where if you merge a transaction that doesn't have an imported_id with an existing transaction that does have an imported_id, the imported_id is removed from the existing transaction.

[strazto]

I'm trying to wrap my head around the difference between #2618 and this PR, but I can't seem to really get it.
Would you mind creating a unit test case that passes in this PR, but would fail in #2618?

I believe the test case that would pass in this PR but not in #2618 would be something like the below. I have tested this test fails in #2618 as the assertion for the number of transactions fails, receiving 2, when expecting 1.

  // ...

Edit: After making this comment I went and got this PR on my machine. The test needed a bit of tweaking to work but in essence asserted my thinking was correct.

@pmoon00 nailed it, thanks for the test case, didn't have time to touch it this weekend :) I'll put it in the PR, can add you to the author list in the release note

I did find an issue/side effect, where if you merge a transaction that doesn't have an imported_id with an existing transaction that does have an imported_id, the imported_id is removed from the existing transaction.

That sounds like a bug, probably the imported_id should be picked over the existing tx.
That said given this PR only touches the selection criteria for the fuzzy search, that bug is not really related to this PR.

Worth raising in its own ticket :)

[MatissJanis]

Ok, this seems to work as expected. Seeing no problems and the extra test case works as expected (fails in master and in the previous PR, but passes here).

LGTM on my end, but I'd like another maintainer to review this too since it touches such an important part of the codebase.