Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

shadow subscriber fatals the server #322

Closed
aricart opened this issue Jun 12, 2017 · 1 comment
Closed

shadow subscriber fatals the server #322

aricart opened this issue Jun 12, 2017 · 1 comment

Comments

@aricart
Copy link
Member

aricart commented Jun 12, 2017

The server for some reason added more than one shadow subscriber to its internal state. This makes it impossible for the server to start.

[12176] 2017/06/12 13:42:29.336373 [INF] STREAM: Failed to start: there should be only one shadow subscriber for ["backend-durable:remove-bubble-group"] queue group panic: there should be only one shadow subscriber for ["backend-durable:remove-bubble-group"] queue group [recovered] panic: there should be only one shadow subscriber for ["backend-durable:remove-bubble-group"] queue group

With some printf, I found:

  • [client id: ]: ID:5 qGroup:"backend-durable:remove-bubble-group" inbox:"_INBOX.M3V34IJY91XSXFTB3G0NIQ" ackInbox:"_INBOX.sYCnOp4GzXNhtBPSPmJr3J" maxInFlight:1 ackWaitInSecs:1800 lastSent:31 isDurable:true
    [12176] 2017/06/12 13:42:29.335988 [INF] STREAM: debug

  • [client id: ]: ID:6 qGroup:"backend-durable:remove-bubble-group" inbox:"_INBOX.F8HZ11KI1TANGB68XPHATU" ackInbox:"_INBOX.bB3pce3nTYX4KGZy9uDjMJ" maxInFlight:1 ackWaitInSecs:1800 lastSent:31 isDurable:true

@aricart
Copy link
Member Author

aricart commented Jun 12, 2017

stan-crashes-on-this-file-storage.tar.gz

To start the server:

nats-streaming-server -cid dev-bh-backend -st FILE -dir <pathToExtractedStoreDir>

kozlovic added a commit that referenced this issue Jun 27, 2017
A shadow queue subscriber is a record left in the store when the last
member of a queue group has left, but the state needs to be maintained.
There should be only one such shadow queue durable subscriber per
queue group per channel.
However, we got a case with a datastore that contained 2 such records.
Code inspection did not reveal how this could happen with current
code base, so it is suspected that this is the result of various
server upgrades with older datastores.
This change prevents the server from panic'ing and instead uses
the shadow subscriber with the highest lastSent value.

Resolves #322
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant