Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JetStream - Updating consumer from R3 to R1 doesn't work #3262

Closed
goku321 opened this issue Jul 14, 2022 · 2 comments · Fixed by #3293
Closed

JetStream - Updating consumer from R3 to R1 doesn't work #3262

goku321 opened this issue Jul 14, 2022 · 2 comments · Fixed by #3293
Assignees

Comments

@goku321
Copy link
Contributor

goku321 commented Jul 14, 2022

nats-server: v2.8.4
nats.go: v1.16.0
NATS cli: v0.0.33

Step to reproduce:

  1. Setup a 3-nodes JetStream cluster.
  2. Create a stream with a replica count 3.
  3. Create a consumer using nats.go client with a replica count 3.
  4. Publish and consume some messages.
  5. Update the consumer's replica count to 1 using nats.go client

Expected results

  1. Replica count for the consumer should get updated to 1.
  2. Consumer should have no replicas or metadata on replica nodes.
  3. nats con info should not show any replicas field for a R1 consumer

Screenshot 2022-07-14 at 12 55 36 PM

  1. When the leader node is killed while consuming, the client should get a "Consumer Migration" message (since, it is R1 now).

Actual results

  1. nats con info does show replica count as 1 (which is correct)
  2. Consumer still have its metadata on other nodes and it's being updated regularly (confirmed with ls -l)
  3. nats con info shows replicas field

Screenshot 2022-07-14 at 12 54 44 PM

  1. When the leader node is killed, the client gets a "Leadership Change" message (that means, it is still > R1)
@matthiashanel
Copy link
Contributor

@goku321 I can confirm that this is not implemented

@goku321
Copy link
Contributor Author

goku321 commented Jul 25, 2022

Thanks @matthiashanel for confirming. As a workaround, we are creating a new R1 consumer with start sequence and copying its config from the existing R3. Here's the example code. To make sure that this R1 resumes from where the old R3 left, we are setting the start sequence of R1 to consumerInfo.Delivered.Stream+1 (of R3). I tested this for a few cases and it seems to be working fine. Would there be any case that this might fail?

matthiashanel added a commit that referenced this issue Jul 27, 2022
fixes #3262

also check peer list on every update

Signed-off-by: Matthias Hanel <mh@synadia.com>
matthiashanel added a commit that referenced this issue Jul 27, 2022
* add implementation for consumer replica change

fixes #3262

also check peer list on every update

Signed-off-by: Matthias Hanel <mh@synadia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants