How does the Reimport logic checks if any incoming Findings match Findings that already exist?

[vini-ppro]

Dear community, we rely heavily on the reimport function.

The documentation says the following:

If any incoming Findings match Findings that already exist, the incoming Findings will be discarded rather than recorded as Duplicates

How is DefectDojo effectively matching incoming vs existent findings? Does it look only on the title attributes of the findings? Does it consider other attributes besides title when doing the match? If yes, what attributes exactly?

We are facing some data inconsistency in DefectDojo and we need to make sure Reimport is working as expected for all our user cases.

Thank you.

[valentijnscholten]

Defect Dojo uses the same logic as for deduplication:

• https://docs.defectdojo.com/en/working_with_findings/finding_deduplication/about_deduplication/
• https://docs.defectdojo.com/en/working_with_findings/finding_deduplication/avoiding_duplicates_via_reimport/

A hash code will be calculated for each finding based on the hash code configuration of the scanner. When the hash codes match, the findings are considered identical. The hash code configuration can be tuned via settings:

django-DefectDojo/dojo/settings/settings.dist.py

Lines 1189 to 1304 in d3cd42c

	# ------------------------------------
	# Hashcode configuration
	# ------------------------------------
	# List of fields used to compute the hash_code
	# The fields must be one of HASHCODE_ALLOWED_FIELDS
	# If not present, default is the legacy behavior: see models.py, compute_hash_code_legacy function
	# legacy is:
	# static scanner: ['title', 'cwe', 'line', 'file_path', 'description']
	# dynamic scanner: ['title', 'cwe', 'line', 'file_path', 'description']
	HASHCODE_FIELDS_PER_SCANNER = {
	# In checkmarx, same CWE may appear with different severities: example "sql injection" (high) and "blind sql injection" (low).
	# Including the severity in the hash_code keeps those findings not duplicate
	"Anchore Engine Scan": ["title", "severity", "component_name", "component_version", "file_path"],
	"AnchoreCTL Vuln Report": ["title", "severity", "component_name", "component_version", "file_path"],
	"AnchoreCTL Policies Report": ["title", "severity", "component_name", "file_path"],
	"Anchore Enterprise Policy Check": ["title", "severity", "component_name", "file_path"],
	"Anchore Grype": ["title", "severity", "component_name", "component_version"],
	"Aqua Scan": ["severity", "vulnerability_ids", "component_name", "component_version"],
	"Bandit Scan": ["file_path", "line", "vuln_id_from_tool"],
	"Burp Enterprise Scan": ["title", "severity", "cwe"],
	"Burp Scan": ["title", "severity", "vuln_id_from_tool"],
	"CargoAudit Scan": ["vulnerability_ids", "severity", "component_name", "component_version", "vuln_id_from_tool"],
	"Checkmarx Scan": ["cwe", "severity", "file_path"],
	"Checkmarx OSA": ["vulnerability_ids", "component_name"],
	"Cloudsploit Scan": ["title", "description"],
	"Coverity Scan JSON Report": ["title", "cwe", "line", "file_path", "description"],
	"SonarQube Scan": ["cwe", "severity", "file_path"],
	"SonarQube API Import": ["title", "file_path", "line"],
	"Sonatype Application Scan": ["title", "cwe", "file_path", "component_name", "component_version", "vulnerability_ids"],
	"Dependency Check Scan": ["title", "cwe", "file_path"],
	"Dockle Scan": ["title", "description", "vuln_id_from_tool"],
	"Dependency Track Finding Packaging Format (FPF) Export": ["component_name", "component_version", "vulnerability_ids"],
	"Horusec Scan": ["title", "description", "file_path", "line"],
	"Mobsfscan Scan": ["title", "severity", "cwe", "file_path", "description"],
	"Tenable Scan": ["title", "severity", "vulnerability_ids", "cwe", "description"],
	"Nexpose Scan": ["title", "severity", "vulnerability_ids", "cwe"],
	# possible improvement: in the scanner put the library name into file_path, then dedup on cwe + file_path + severity
	"NPM Audit Scan": ["title", "severity", "file_path", "vulnerability_ids", "cwe"],
	"NPM Audit v7+ Scan": ["title", "severity", "cwe", "vuln_id_from_tool"],
	# possible improvement: in the scanner put the library name into file_path, then dedup on cwe + file_path + severity
	"Yarn Audit Scan": ["title", "severity", "file_path", "vulnerability_ids", "cwe"],
	# possible improvement: in the scanner put the library name into file_path, then dedup on vulnerability_ids + file_path + severity
	"Mend Scan": ["title", "severity", "description"],
	"ZAP Scan": ["title", "cwe", "severity"],
	"Qualys Scan": ["title", "severity", "endpoints"],
	# 'Qualys Webapp Scan': ['title', 'unique_id_from_tool'],
	"PHP Symfony Security Check": ["title", "vulnerability_ids"],
	"Clair Scan": ["title", "vulnerability_ids", "description", "severity"],
	# for backwards compatibility because someone decided to rename this scanner:
	"Symfony Security Check": ["title", "vulnerability_ids"],
	"DSOP Scan": ["vulnerability_ids"],
	"Acunetix Scan": ["title", "description"],
	"Terrascan Scan": ["vuln_id_from_tool", "title", "severity", "file_path", "line", "component_name"],
	"Trivy Operator Scan": ["title", "severity", "vulnerability_ids", "description"],
	"Trivy Scan": ["title", "severity", "vulnerability_ids", "cwe", "description"],
	"TFSec Scan": ["severity", "vuln_id_from_tool", "file_path", "line"],
	"Snyk Scan": ["vuln_id_from_tool", "file_path", "component_name", "component_version"],
	"GitLab Dependency Scanning Report": ["title", "vulnerability_ids", "file_path", "component_name", "component_version"],
	"SpotBugs Scan": ["cwe", "severity", "file_path", "line"],
	"JFrog Xray Unified Scan": ["vulnerability_ids", "file_path", "component_name", "component_version"],
	"JFrog Xray On Demand Binary Scan": ["title", "component_name", "component_version"],
	"Scout Suite Scan": ["file_path", "vuln_id_from_tool"], # for now we use file_path as there is no attribute for "service"
	"Meterian Scan": ["cwe", "component_name", "component_version", "description", "severity"],
	"Github Vulnerability Scan": ["title", "severity", "component_name", "vulnerability_ids", "file_path"],
	"Solar Appscreener Scan": ["title", "file_path", "line", "severity"],
	"pip-audit Scan": ["vuln_id_from_tool", "component_name", "component_version"],
	"Rubocop Scan": ["vuln_id_from_tool", "file_path", "line"],
	"JFrog Xray Scan": ["title", "description", "component_name", "component_version"],
	"CycloneDX Scan": ["vuln_id_from_tool", "component_name", "component_version"],
	"SSLyze Scan (JSON)": ["title", "description"],
	"Harbor Vulnerability Scan": ["title", "mitigation"],
	"Rusty Hog Scan": ["file_path", "payload"],
	"StackHawk HawkScan": ["vuln_id_from_tool", "component_name", "component_version"],
	"Hydra Scan": ["title", "description"],
	"DrHeader JSON Importer": ["title", "description"],
	"Whispers": ["vuln_id_from_tool", "file_path", "line"],
	"Blackduck Hub Scan": ["title", "vulnerability_ids", "component_name", "component_version"],
	"Veracode SourceClear Scan": ["title", "vulnerability_ids", "component_name", "component_version", "severity"],
	"Vulners Scan": ["vuln_id_from_tool", "component_name"],
	"Twistlock Image Scan": ["title", "severity", "component_name", "component_version"],
	"NeuVector (REST)": ["title", "severity", "component_name", "component_version"],
	"NeuVector (compliance)": ["title", "vuln_id_from_tool", "description"],
	"Wpscan": ["title", "description", "severity"],
	"Popeye Scan": ["title", "description"],
	"Nuclei Scan": ["title", "cwe", "severity", "component_name"],
	"KubeHunter Scan": ["title", "description"],
	"kube-bench Scan": ["title", "vuln_id_from_tool", "description"],
	"Threagile risks report": ["title", "cwe", "severity"],
	"Trufflehog Scan": ["title", "description", "line"],
	"Humble Json Importer": ["title"],
	"MSDefender Parser": ["title", "description"],
	"HCLAppScan XML": ["title", "description"],
	"HCL AppScan on Cloud SAST XML": ["title", "file_path", "line", "severity"],
	"KICS Scan": ["file_path", "line", "severity", "description", "title"],
	"MobSF Scan": ["title", "description", "severity"],
	"MobSF Scorecard Scan": ["title", "description", "severity"],
	"OSV Scan": ["title", "description", "severity"],
	"Snyk Code Scan": ["vuln_id_from_tool", "file_path"],
	"Deepfence Threatmapper Report": ["title", "description", "severity"],
	"Bearer CLI": ["title", "severity"],
	"Nancy Scan": ["title", "vuln_id_from_tool"],
	"Wiz Scan": ["title", "description", "severity"],
	"Kubescape JSON Importer": ["title", "component_name"],
	"Kiuwan SCA Scan": ["description", "severity", "component_name", "component_version", "cwe"],
	"Rapplex Scan": ["title", "endpoints", "severity"],
	"AppCheck Web Application Scanner": ["title", "severity"],
	"AWS Inspector2 Scan": ["title", "severity", "description"],
	"Legitify Scan": ["title", "endpoints", "severity"],
	"ThreatComposer Scan": ["title", "description"],
	"Invicti Scan": ["title", "description", "severity"],
	"Checkmarx CxFlow SAST": ["vuln_id_from_tool", "file_path", "line"],
	"HackerOne Cases": ["title", "severity"],
	"KrakenD Audit Scan": ["description", "mitigation", "severity"],
	"Red Hat Satellite": ["description", "severity"],
	"Qualys Hacker Guardian Scan": ["title", "severity", "description"],
	}

For findings with endpoints these fields are considered to deduplicate endpoints:

django-DefectDojo/dojo/settings/settings.dist.py

Lines 1405 to 1414 in d3cd42c

	# Allows to deduplicate with endpoints if endpoints is not included in the hashcode.
	# Possible values are: scheme, host, port, path, query, fragment, userinfo, and user. For a details description see https://hyperlink.readthedocs.io/en/latest/api.html#attributes.
	# Example:
	# Finding A and B have the same hashcode. Finding A has endpoint http://defectdojo.com and finding B has endpoint https://defectdojo.com/finding.
	# - An empyt list ([]) means, no fields are used. B is marked as duplicated of A.
	# - Host (['host']) means: B is marked as duplicate of A because the host (defectdojo.com) is the same.
	# - Host and path (['host', 'path']) means: A and B stay untouched because the path is different.
	#
	# If a finding has more than one endpoint, only one endpoint pair must match to mark the finding as duplicate.
	DEDUPE_ALGO_ENDPOINT_FIELDS = ["host", "path"]

• https://docs.defectdojo.com/en/open_source/installation/configuration/

If applicable, please mark the correct answer as "Answer".