Deleted transactions are imported again with same external id

[james-prince]

Support guidelines

• I've read the support guidelines

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem
• ... it's definitely a Firefly III issue, not me

Description

When running a gocardless import using external identifier based duplicate detection, a deleted transaction is imported again with the same external identifier and other information.

Debug information

Debug information generated at 2025-04-02 23:26:22 for Firefly III version v6.2.10.

System information
Item	Value
Firefly III	v6.2.10 / #25 (expects #25)
PHP version	8.4.4 (64bits) / fpm-fcgi / Linux x86_64
BCscale	12
Error reporting	Display: Off, reporting: ALL errors
Max upload	104857600 (100 MB)
Database drivers	*mysql*, pgsql, sqlite,

Firefly III information
Item	Value
Timezone	Europe/London + Europe/London
App environment	production, debug: false
Layout	v1
Logging	notice, stdout / (empty)
Cache driver, session driver	file, file
Default language and locale	en_US + equal
Trusted proxies	**
Login provider & user guard	eloquent / remote_user_guard
Login headers	HTTP_X_AUTHENTIK_EMAIL +
Stateful domains
Last cron job	2025-04-01 23:00:00 (23 hours ago)
Mailer	log
Exchange rates	Disabled, downloads disabled
RB-column	Disabled

User-specific information
Item	Value
User	#1 of 1
User flags	💳 🐷 📒 🔧 🕜 📑 📧
Native currency	GBP
Convert to native currency?	Disabled
Session start	2025-04-01 00:00:00
Session end	2025-04-30 23:59:59
View range	1M
User language	en_GB
User locale	en_GB
Locale(s) supported	en_GB.utf8: ✅ en_GB.UTF-8: ✅
User agent	Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:136.0) Gecko/20100101 Firefox/136.0

Expected behaviour

The deleted transaction should not be imported again

Steps to reproduce

1. Run a gocardless import
2. Pick a transaction and delete it
3. Run the gocardless import again
4. The transaction has been created again

Additional info

The contents of my gocardless json file:

{
    "version": 3,
    "source": "ff3-importer-1.6.1",
    "created_at": "2025-04-01T01:14:55+01:00",
    "date": "",
    "default_account": 1,
    "delimiter": "comma",
    "headers": false,
    "rules": true,
    "skip_form": false,
    "add_import_tag": true,
    "roles": [],
    "do_mapping": [],
    "mapping": [],
    "duplicate_detection_method": "cell",
    "ignore_duplicate_lines": false,
    "unique_column_index": 0,
    "unique_column_type": "external-id",
    "flow": "nordigen",
    "content_type": "csv",
    "custom_tag": "",
    "identifier": "0",
    "connection": "0",
    "ignore_spectre_categories": false,
    "grouped_transaction_handling": "",
    "use_entire_opposing_address": false,
    "map_all_data": false,
    "accounts": {
        "REDACTED": 6,
        "REDACTED": 7
    },
    "date_range": "partial",
    "date_range_number": 30,
    "date_range_unit": "d",
    "date_not_before": "2025-03-02",
    "date_not_after": "",
    "nordigen_country": "GB",
    "nordigen_bank": "REDACTED",
    "nordigen_requisitions": {
        "REDACTED": "REDACTED"
    },
    "nordigen_max_days": "90",
    "conversion": false,
    "ignore_duplicate_transactions": false
}

[james-prince]

Just some further info, i can see all of the deleted transactions inside the DB still (as well as the last imported non-deleted transaction).

Is there somewhere i can see the external identifiers for each row inside the DB? I couldn't find a column with that information inside for the transactions / their related tables.

[JC5]

Hey, thanks for opening an issue.

I'm surprised that you run into this issue, because I also use external ID matching and it works for me ™ . I delete transactions regularly and they never get imported again.

• Could you validate in the journal_meta table what the external ID values are?
• Could you share some logs of two runs, where Firefly III imports a duplicate entry?

Cheers,
James

[james-prince]

Here are the journal_meta records with the same external id:

And here is the log during the import, hopefully there is enough info:
ff3-import-log.md

[JC5]

Mmm, i see nothing special in the table.

Do you have the logs from the data importer itself? They should also contain debug entries.

[JC5]

Here is an example of what I would like to see:

[2025-04-03 14:26:30] production.DEBUG: Now submitting transaction 60/99  
[2025-04-03 14:26:30] production.DEBUG: Going to search for external_id:fd8cfc78a7be0f2b47c5d15a6fafc500 using query external_id_is:"fd8cfc78a7be0f2b47c5d15a6fafc500"  
[2025-04-03 14:26:30] production.DEBUG: No base url in getBaseUrl() session, will return config variable.  
[2025-04-03 14:26:30] production.DEBUG: No access token in hasAccessToken() session, will return config variable.  
[2025-04-03 14:26:30] production.DEBUG: Found 1 transaction(s). Return group ID #19897.  

[james-prince]

Sorry i thought i'd enabled debug logging, I'm rate limited currently but i'll share the logs again once the limit resets

[JC5]

Not a problem.

Could you, in your configuration, set "ignore_duplicate_transactions": true?

I believe that will solve it.

[james-prince]

Hi James, unfortunately it seems that this didn't work:
"ignore_duplicate_transactions": true

Managed to get the logs correctly this time, let me know if you need any further info

[2025-04-04 01:02:37] local.DEBUG: Going to search for external_id:1a74bfef-0a33-36dc-8686-2acd96da91b4 using query external_id_is:"1a74bfef-0a33-36dc-8686-2acd96da91b4"
[2025-04-04 01:02:37] local.DEBUG: Looks like field "external_id" with value "1a74bfef-0a33-36dc-8686-2acd96da91b4" is unique, return false.
[2025-04-04 01:02:37] local.DEBUG: Transaction #1 is unique.
[2025-04-04 01:02:37] local.DEBUG: Going to map data for this line.
[2025-04-04 01:02:37] local.DEBUG: Submitting to Firefly III: {"apply_rules":true,"error_if_duplicate_hash":false,"transactions":[{"type":"withdrawal","date":"2025-03-27T00:00:00+00:00","amount":"0.050000000000","description":"REDACTED","payment_date":"","order":0,"currency_code":"GBP","tags":["booked"],"category_name":null,"category_id":null,"notes":"","external_id":"1a74bfef-0a33-36dc-8686-2acd96da91b4","internal_reference":"","additional-information":"","bonus_tags":[],"source_id":6,"destination_number":"","destination_name":"(unknown destination account)"}]}
[2025-04-04 01:02:37] local.DEBUG: Now at App\Http\Controllers\Import\SubmitController::status(submission-4lD3cu2dwghB)
[2025-04-04 01:02:37] local.DEBUG: Now in startOrFindJob(submission-4lD3cu2dwghB)
[2025-04-04 01:02:37] local.DEBUG: Try to see if file exists for job submission-4lD3cu2dwghB.
[2025-04-04 01:02:37] local.DEBUG: Status file exists for job submission-4lD3cu2dwghB.
[2025-04-04 01:02:37] local.DEBUG: Status: submission_running

It also only performs the import again once the transaction is deleted, if i leave the transaction there its not imported again:

[2025-04-04 01:25:37] local.DEBUG: Now submitting transaction 1/1
[2025-04-04 01:25:37] local.DEBUG: Going to search for external_id:1a74bfef-0a33-36dc-8686-2acd96da91b4 using query external_id_is:"1a74bfef-0a33-36dc-8686-2acd96da91b4"
[2025-04-04 01:25:37] local.DEBUG: Found 1 transaction(s). Return group ID #2477.
[2025-04-04 01:25:37] local.DEBUG: Looks like field "external_id" with value "1a74bfef-0a33-36dc-8686-2acd96da91b4" is not unique, found in group #2477. Return false
[2025-04-04 01:25:37] local.DEBUG: Transaction #1 is NOT unique.
[2025-04-04 01:25:37] local.INFO: Done submitting 1 transactions to your Firefly III instance.
[2025-04-04 01:25:37] local.INFO: Actually imported and not duplicate: 0.
[2025-04-04 01:25:37] local.DEBUG: Routine manager: messages: 0, warnings: 0, errors: 0

Cheers,
James

[JC5]

"error_if_duplicate_hash":false,

This should be true when ignore_duplicate_transactions is true. You can try setting ignore_duplicate_lines to true as well, but it should not matter.

[james-prince]

I've tried setting these values to true and still get the same behaviour:

{
"version": 3,
"source": "ff3-importer-1.6.1",
"created_at": "2025-04-01T01:14:55+01:00",
"date": "",
"default_account": 1,
"delimiter": "comma",
"headers": false,
"rules": true,
"skip_form": false,
"add_import_tag": true,
"roles": [],
"do_mapping": [],
"mapping": [],
"duplicate_detection_method": "cell",
"ignore_duplicate_lines": true,
"unique_column_index": 0,
"unique_column_type": "external-id",
"flow": "nordigen",
"content_type": "csv",
"custom_tag": "",
"identifier": "0",
"connection": "0",
"ignore_spectre_categories": false,
"grouped_transaction_handling": "",
"use_entire_opposing_address": false,
"map_all_data": false,
"accounts": {
"REDACTED": 6,
"REDACTED": 7
},
"date_range": "partial",
"date_range_number": 30,
"date_range_unit": "d",
"date_not_before": "2025-03-02",
"date_not_after": "",
"nordigen_country": "GB",
"nordigen_bank": "REDACTED",
"nordigen_requisitions": {
"REDACTED": "REDACTED"
},
"nordigen_max_days": "90",
"conversion": false,
"ignore_duplicate_transactions": true,
"error_if_duplicate_hash": true
}

Interestingly, the log still reports "error_if_duplicate_hash":false despite it being true in my json config file

[2025-04-04 12:05:23] local.DEBUG: Now submitting transaction 1/1
[2025-04-04 12:05:23] local.DEBUG: Going to search for external_id:1a74bfef-0a33-36dc-8686-2acd96da91b4 using query external_id_is:"1a74bfef-0a33-36dc-8686-2acd96da91b4"
[2025-04-04 12:05:23] local.DEBUG: Looks like field "external_id" with value "1a74bfef-0a33-36dc-8686-2acd96da91b4" is unique, return false.
[2025-04-04 12:05:23] local.DEBUG: Transaction #1 is unique.
[2025-04-04 12:05:23] local.DEBUG: Going to map data for this line.
[2025-04-04 12:05:23] local.DEBUG: Submitting to Firefly III: {"apply_rules":true,"error_if_duplicate_hash":false,"transactions":[{"type":"withdrawal","date":"2025-03-27T00:00:00+00:00","amount":"0.050000000000","description":"REDACTED","payment_date":"","order":0,"currency_code":"GBP","tags":["booked"],"category_name":null,"category_id":null,"notes":"","external_id":"1a74bfef-0a33-36dc-8686-2acd96da91b4","internal_reference":"","additional-information":"","bonus_tags":[],"source_id":6,"destination_number":"","destination_name":"(unknown destination account)"}]}

Feels like the issue lies here, that the value isn't being detected as a duplicate:

[2025-04-04 12:05:23] local.DEBUG: Looks like field "external_id" with value "1a74bfef-0a33-36dc-8686-2acd96da91b4" is unique, return false.

[diamant-x]

HI, I'm facing the same issue since upgrading FiDi to 1.6.1.
I think the journal_meta or the journal_id creation is bugged. More info below:

before re-attempting the import. journal_meta:

The last one i proceed to delete before the import.
Then I run the import (Can't get the logs I'll edit if I can but meanwhile hope this can help).
FiDi on-screen logs:

Those last 2 entries are the ones that are getting reimported but that were deleted. Interestingly enough other transactions check work fine as seen in the top lines, but seems like for deleted only some checks may be failing.
after the re-import. journal_meta:

Interestingly enough, I see that the 'new transaction' has somethign wrong:

• the newly created transaction id 12981. When I look in FFIII webui it shows external_id: 977e39f87a91ae9386d88a25b4c8cd3b
  

• However, when I search in journal_met for that transaction id, it shows external_id: b0d4c2aca25fe4939be71e7b792127fc and looks like a different transaction all together based on the dates.
  

This started in the last update of FiDi, in case it helps to circle down to changes. I've been a long-term user and this was not a problem before (so config is fine).

[JC5]

I've not yet been able to replicate this. Could you share your logs and your configuration, please?

[diamant-x]

I've not yet been able to replicate this. Could you share your logs and your configuration, please?

I'll try. I'm at the moment having problems because i'm using the HAOS addon and the env variables get autoconverted to quotes. so i set in FiDI environment variables:

APP_DEBUG: true
LOG_LEVEL: debug
LOG_CHANNEL: stack

and these get translated to:

APP_DEBUG='true'
LOG_LEVEL='debug'
LOG_CHANNEL='stack'

which I guess then doesn't enable the debugging properly as it needs to go without quotes?
I was checking the 'debug' mode of the transactions though and I can see the journal_id is different in these places. Is this normal?

[JC5]

Yes, that's normal.

[JC5]

All I can do right now is add some debug information.

If:

"ignore_duplicate_transactions": true

Is true, the data importer should set error_if_duplicate_hash=true and all should be well.

If you import with the latest develop tag, the logs should report the value of error_if_duplicate_hash.

[diamant-x]

Since installing the last versions of develop (currently using 20250408 (19-04-2025)) issue is no longer occurring. ＼（〇_ｏ）／