Spectre importer not detecting some duplicates

[Ameb]

Bug description

Spectre import duplicates some transactions. I ran the importer some days ago, and today i ran it again and some transactions (9 out of 120) weren't detected as duplicates. All of them are within a week (22/12 - 28/12).

I'm attaching some of the transactions using the /transactions/debug/xxx trick. They have some censored strings.
dx_yyy.json where x is the duplicated transaction and y the transaction id. Github doesn't allow json attachments so i have ziped them.

Steps to reproduce

Only happens for some transactions. Hard to reproduce.

Extra info

• The transactions are not duplicated in Salt Edge
• They have the same internal_reference data
• Duplicated transactions dates are in a row

Debug information generated at 2021-02-28 00:42:06 Europe/Madrid for Firefly III version 5.4.6.

Scope	Version
Firefly III	5.4.6
Firefly III API	1.4.0
PHP	7.4.14
Host	Linux

System info	Value
Installation ID	ac5ebab1-0eab-4b00-ab79-387aeb9efdf7
Using docker?	true
Telemetry	false
Layout	v1
App environment	local
App debug mode	true
App cache driver	file
App logging	info, stack
Display errors	Off
Error reporting	ALL errors
Interface	apache2handler
Default language	en_US
Default locale	es_ES
BCscale	12
DB drivers	mysql, pgsql, sqlite
Current driver	mysql
DB version	15 (exp. 15)
Login provider	eloquent
Trusted proxies (.env)	**

User info	Value
Session start	2020-12-01 00:00:00
Session end	2020-12-31 00:00:00
Session first	2019-02-01 00:00:00
User ID	1
User language	en_GB
User locale	en_GB
Attempt at "en_GB.utf8"	'en_GB.utf8'
Attempt at "en_GB.UTF-8"	'en_GB.UTF-8'
User agent	Mozilla/5.0 (X11; Linux x86_64; rv:86.0) Gecko/20100101 Firefox/86.0

Bonus points

duplicates.zip

[JC5]

It looks like the transactions have a different external ID. You can compare the JSON files yourself on JSON compare.

[Ameb]

I did, but didn't understood some properties. I was unlucky to check 3 transactions on 3 different accounts so I made wrong assumptions about the meaning of internal_reference (it is the account id on salt edge).

Thank you the awesome app and your time!

[Ameb]

TLDR: Salt Edge got new IDs for some of my transactions.

Manually deleting the original transactions should fix it (shouldn't happen often).
Checking the downloaded file by the spectre importer on /var/www/html/storage/downloads (importer filesystem) helped to troubleshoot and undestand what data was being loaded into the app.

[Ameb]

Those docs also apply:

https://docs.firefly-iii.org/csv/errors/duplicate_transactions/