Deduplication of queued CIDs is limited by the size of the deduplication cache. If the cache is size 2048, then CIDs are only deduplicated within the last 2048 queued.
If CID duplication is more sparse that the cache size, or if a small number duplicate CIDs do not cause any issues or take up significant space, then the deduplication cache is only consuming memory with little or no benefit.
Deduplication cache size defined here
Deduplication of queued CIDs is limited by the size of the deduplication cache. If the cache is size 2048, then CIDs are only deduplicated within the last 2048 queued.
If CID duplication is more sparse that the cache size, or if a small number duplicate CIDs do not cause any issues or take up significant space, then the deduplication cache is only consuming memory with little or no benefit.
Deduplication cache size defined here