Duplicates after Re-Scan

[Psychi1]

Hello,

I put the same document (by contents) to paperless, although they are already in the database (by an former scan). Duplicates are only detected if they are really the same documents by hashing. However sometimes you accidentally do a re-scan, which leads to another hash and results in an undetected duplicate.

So I propose an on-demand task to detect these duplicated by comparing correspondent, date (created as written in the doc) and page count. I know there are some weak points. The first point is, you need to do all the work to validate the meta-data and second, non-duplicates will marked as possible duplicates again, when you start the task again.

[donnyxray]

I also end up with duplicates, though for a different reason. Many of the documents I receive are generated on-the-fly by the systems sending them. When such documents are sent multiple times (e.g. reminder invoices), each copy has a different md5 hash, even when the contents is identical to the human eye.

Hence I agree that the hash method is not always suitable. In my case looking at the filename on a per-correspondent basis would fix the issue. Ideally Paperless-NGX allows us to enable additional matching methods, both globally as well as on a per-correspondent basis.

[flesser]

I also would love to see something like a "soft duplicate" detection, e.g. based on text content or similar criteria, for re-scans of the same document. Either as a dedicated page view / filter action or as some kind of hint during document editing. (e.g. "This document looks like a duplicate of document #NNN").

[flemmingss]

I agree.
This should be implemented as a warning where you have a three choices:

• This is not a duplicate, keep both
• Keep "A", and delete "B"
• Keep "B", and delete "A"

[stumpylog]

The new fuzzy matching management tool #4160 does this. It cannot be done at consume time, as that is an O(n^2) operation across all documents and simply not a feasible task to accomplish.

[github-actions[bot]]

This discussion has been automatically closed because it was marked as answered.

[github-actions[bot]]

This discussion has been automatically locked since there has not been any recent activity after it was closed. Please open a new discussion for related concerns.