Messages not deleted from WorkQueuePolicy stream when replicas > 1

[danpoland]

It may be that I misunderstand the meaning of the Replicas configuration for a stream. For demo purposes, I have a NATS cluster with two servers running JetStream.

First I created a stream called FOO using the Work Queue retention policy and set the number of replicas to 2.

Stream FOO was created

Information for Stream FOO created 2021-05-21T15:32:16-04:00

Configuration:

             Subjects: FOO.*
     Acknowledgements: true
            Retention: File - WorkQueue
             Replicas: 2
       Discard Policy: Old
     Duplicate Window: 2m0s
     Maximum Messages: unlimited
        Maximum Bytes: unlimited
          Maximum Age: 0.00s
 Maximum Message Size: unlimited
    Maximum Consumers: unlimited


Cluster Information:

                 Name: local
               Leader: nats1
              Replica: nats2, current, seen 0.00s ago

State:

             Messages: 0
                Bytes: 0 B
             FirstSeq: 0
              LastSeq: 0
     Active Consumers: 0

I then created a pull-based consumer called TEST, that pulls from the FOO.test subject.

Information for Consumer FOO > TEST created 2021-05-21T15:36:52-04:00

Configuration:

        Durable Name: TEST
           Pull Mode: true
      Filter Subject: FOO.test
         Deliver All: true
          Ack Policy: Explicit
            Ack Wait: 30s
       Replay Policy: Instant
     Max Ack Pending: 20,000
        Flow Control: false

Cluster Information:

                Name: local
              Leader: nats2
             Replica: nats1, current, seen 0.01s ago

State:

   Last Delivered Message: Consumer sequence: 0 Stream sequence: 0
     Acknowledgment floor: Consumer sequence: 0 Stream sequence: 0
         Outstanding Acks: 0 out of maximum 20000
     Redelivered Messages: 0
     Unprocessed Messages: 0

I then published a single message to the FOO.test subject and confirmed it was sitting in the stream and was unprocessed by the consumer.

❯ nats s report
Obtaining Stream stats

+----------------------------------------------------------------------------------+
|                                  Stream Report                                   |
+--------+---------+-----------+----------+-------+------+---------+---------------+
| Stream | Storage | Consumers | Messages | Bytes | Lost | Deleted | Replicas      |
+--------+---------+-----------+----------+-------+------+---------+---------------+
| FOO    | File    | 1         | 1        | 43 B  | 0    | 0       | nats1*, nats2 |
+--------+---------+-----------+----------+-------+------+---------+---------------+

❯ nats c report
? Select a Stream FOO
Consumer report for FOO with 1 consumers

+----------+------+------------+----------+-------------+-------------+-------------+-----------+---------------+
| Consumer | Mode | Ack Policy | Ack Wait | Ack Pending | Redelivered | Unprocessed | Ack Floor | Cluster       |
+----------+------+------------+----------+-------------+-------------+-------------+-----------+---------------+
| TEST     | Pull | Explicit   | 30.00s   | 0           | 0           | 1 / 100%    | 0         | nats1, nats2* |
+----------+------+------------+----------+-------------+-------------+-------------+-----------+---------------+

Finally, I pulled the message from the TEST consumer and acknowledged it. The message is removed from the consumer but remains on the stream.

❯ nats con next FOO TEST
[15:40:11] subj: FOO.test / tries: 1 / cons seq: 1 / str seq: 1 / pending: 0

hello

Acknowledged message

❯ nats c report
? Select a Stream FOO
Consumer report for FOO with 1 consumers

+----------+------+------------+----------+-------------+-------------+-------------+-----------+---------------+
| Consumer | Mode | Ack Policy | Ack Wait | Ack Pending | Redelivered | Unprocessed | Ack Floor | Cluster       |
+----------+------+------------+----------+-------------+-------------+-------------+-----------+---------------+
| TEST     | Pull | Explicit   | 30.00s   | 0           | 0           | 0           | 1         | nats1, nats2* |
+----------+------+------------+----------+-------------+-------------+-------------+-----------+---------------+

❯ nats s report
Obtaining Stream stats

+----------------------------------------------------------------------------------+
|                                  Stream Report                                   |
+--------+---------+-----------+----------+-------+------+---------+---------------+
| Stream | Storage | Consumers | Messages | Bytes | Lost | Deleted | Replicas      |
+--------+---------+-----------+----------+-------+------+---------+---------------+
| FOO    | File    | 1         | 1        | 43 B  | 0    | 0       | nats1*, nats2 |
+--------+---------+-----------+----------+-------+------+---------+---------------+

I expected the message to also be removed from the stream since this is a work queue stream. I confirmed that the message is removed from the stream when replicas is set to 1.

Is this a bug or am I misinterpreting the replicas configuration?

Thanks for your help.

[ripienaar]

I will investigate, but do not create R2 streams pls

[ripienaar]

I can confirm I can reproduce this one

[danpoland]

I will investigate, but do not create R2 streams pls

Yup, the 2 node cluster was just something I set up locally to demo. In production, I am using a cluster of three nodes (which has the same issue.)