Deduplication MsgId are not deleted during Purge operation

[funkye]

Observed behavior

We have a stream with deduplication. When we publish messages with different MsgIds and a specific subject, then call Purge on the subject, Nats server info says that there are 0 messages in the stream, but next messages with the same MsgIds are not published due to deduplication. Also, with long-term use of deduplication and Purge operations, it is noticeable that the memory consumed by the server increases, even with a small number of remaining messages. After restarting the server, the memory stabilizes and deduplication does not work on deleted messages ids.

Expected behavior

After cleaning messages with Purge operation, we can publish messages with MsgIds of deleted messages. Memory from storing MsgIds should be cleared when messages are deleted.

Server and client version

Tested server versions: 2.10.7, 2.10.18, 2.10.27, 2.11.1
.NET client version: 2.5.8

Host environment

Windows 11

Steps to reproduce

1. Publishing a message with a given subject and MsgId to a stream with the deduplication window enabled.
2. Cleaning the stream for a given subject.
3. Republishing the message with the same parameters.

[neilalexander]

For a full stream purge this is simple enough, we can just blow away the ddmap and ddarr.

If it's a partial purge (i.e. up to a sequence number) then this becomes a lot more expensive, since we have to then iterate over the whole ddmap looking for individual sequences and then re-stitch the entire ddarr, which depending on the publish rate, could be quite large.

This is even worse if it's a purge by subject, since the filestore doesn't surface the purged sequences back to the stream except for via the storage update handler, and the deduplication map doesn't track subjects.

@derekcollison I know we've talked in the past about rewriting the deduplication tracking altogether, and IIRC we had one attempt at doing so but it was backed out. Maybe time to revisit it?

[MauriceVanVeen]

For the time being you could also look into using MaxMsgsPerSubject=1 with DiscardNew per subject. That would mean all your subjects need to be unique, but they can be used for de-duplication that way as well. If at any point you remove or purge a message, then that subject/dedupe-id is usable again for new data.

[derekcollison]

This is a tough one, meaning whether or not purge should wipe a de-dupe state or not.

[funkye]

Hi! Thanks for the advice. If we take the solution with 1 message per subject, how much will it theoretically load the server if millions of subjects are created and deleted per day? Will this approach work in theory at high loads?

[MauriceVanVeen]

A stream can handle millions of subjects. It depends more on how many subjects your stream will hold at any given time. So let's say you publish a million messages with unique subjects every day, but you also delete them, such that you only ever hold 250k messages and unique subjects at a given time.

Best to test it out and see what total you'll reach in terms of stream size. It has the added benefit however that you can do deduplication for way longer than you'd normally make the dedupe window, making it more resilient/reliable. But also supports this case of, if you purge the stream, you can publish (duplicate) subjects again, because they are gone.