Deduplication dedupe the oldest vulnerability

[SeekNHack]

Be informative
I am running DefectDojo on Docker via Portainer. I set a env var to duplicate my test 'Pen Test' using Unique Id From Tool (that I calculate using a self-made python library to use API). My Unique Id is calculated by: Endpoint (host,port,protocol), IP Address and Vulnerability Title

Bug description
Duplication seems work, but DefectDojo set as "Duplicate" the oldest vulnerability (with a lower id and date) instead of the newest.

Expected behavior
The newest vulnerability is marked as "Duplicated"

Deployment method (select with an X)

• [x ] Docker Compose
• Kubernetes
• GoDojo

Environment information

• Operating System: Debian GNU/Linux 12 (bookworm)
• Docker Compose version: 1
• DefectDojo version (see footer): v. 2.43.3 ( release mode )

Screenshots
In this example, they have same UNIQUE ID FROM TOOLS but some differences about Severity, Date and descriptions. They are in 2 different Engagements.

How can I fix this?

[valentijnscholten]

Can you elaborate the exact steps that you take? Dit you create your own parser which sets the unique_id_from tool and then you see the behaviour? Or are you manipulating the unique_id_from tool via the API and using an existing parser/scantype?

[SeekNHack]

Yes

Creation of Product, Engagement, Test and Finding via API

First of all, I created via API a product, engagement, and a test (test type named Pen Test) and uploaded via API 2 findings.

Set up custom parser

Then, I set up in my env this variable:
DD_DEDUPLICATION_ALGORITHM_PER_PARSER: '{"Pen Test": "unique_id_from_tool"}' in uwsgi and both celerys.

Deduplication test

I created a new finding (it's the info one in the screenshot) in a different Engagement, with a test type Pen Test into it with the same Unique ID (inserted manually, not via API to test the duplication) then, I run this command on docker:

docker exec -it defectdojo-uwsgi-1 ./manage.py dedupe --parser "Pen Test" --dedupe_only

After 5 minutes, I found the oldest vulnerability was checked as Duplicated, like in the screenshot

[valentijnscholten]

@SeekNHack Could you try again after applying the changes from https://github.com/DefectDojo/django-DefectDojo/pull/11964/files ?

[SeekNHack]

@valentijnscholten It works!

[valentijnscholten]

@SeekNHack Turns out that the ordering was already correct even without the PR. Could it be that you run into a timing issue? Dedupe happens in the background so even with the correct ordering, there could be small timing changes, especially if you have multiple celery workers.