Improve duplicates handling

[crtxcr]

Currently we won't add the same file twice simply by matching the path, but if there is a copy somewhere else, we will add that one too and also reindex it.

Thus, either:

• Don't store duplicates
• At least reuse the index.

and:
Maybe an option to prune duplicates afterwards.