Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

`NATS: Subject must be supplied` when `msg.ack` #115

Open
wojtkowiak opened this issue Jun 18, 2019 · 12 comments

Comments

Projects
None yet
3 participants
@wojtkowiak
Copy link

commented Jun 18, 2019

I am getting this very weird situation in which sometimes I get NATS: Subject must be supplied when ack'ing a message. It goes away after restart. The weird part is that the connection is stable and I do receive messages on the subscription. But when this happens the instance cannot ack a message any longer, it always fail with this until I restart it.
Any idea what my be causing this?

@aricart

This comment has been minimized.

Copy link
Member

commented Jun 18, 2019

@wojtkowiak the error is coming from the underlying nats library.

I think I partly understand the issue:

  • Your client is sending out a subscription
  • The server accepts it
  • The subscription response is not returning to the client (timeout should be emitted)
  • If timeout was emitted, the client cannot ack, so it needs to ignore any messages from the suscription, and close() or unsubscribe()

The error you are getting is because ack() depends on the subscription response that it never got.

The part that I am puzzled about is that you say the subscription was working normally (ie, you were receiving and ack'ing messages). I wouldn't have expected the subscription to work.

Could you:

  • Add a timeout, error, ready handlers to the subscription
  • Add a connection_lost to the stan connection
@aricart

This comment has been minimized.

Copy link
Member

commented Jun 20, 2019

@wojtkowiak any updates from your side?

@aricart

This comment has been minimized.

Copy link
Member

commented Jun 21, 2019

@wojtkowiak Can you please report the version of the server. @kozlovic can sub protocol messages be published by a different connection than subscription messages?

@wojtkowiak

This comment has been minimized.

Copy link
Author

commented Jun 24, 2019

sorry for stalling this, I am using nats:1.4.0, I will be adding some more logs and check out if I do listen to every events. Will get back on that.

Your client is sending out a subscription
The server accepts it
The subscription response is not returning to the client (timeout should be emitted)
If timeout was emitted, the client cannot ack, so it needs to ignore any messages from the subscription, and close() or unsubscribe()

From the logs I can see that subscription is opened correctly, I do receive messages, the processing usually takes no more than 2-5sec, ackTime is set to 20000. Then out of a sudden I get the error and like I said, I still do receive next messages but they fail in ack, I checked the timestamps and there was 1sec delay between receiving and acking message.

@nazar-pc

This comment has been minimized.

Copy link

commented Jul 15, 2019

I see the same issue in our setup.

NATS version: 1.4.1
NATS Streaming: 2.0.0
node-nats-streaming: 0.2.2

At some point some instances start getting this error and as was said before, there is no way to get rid of it until restart.

Also I looked as source code and it feels like there is nothing in JS library that may cause this.
Subject comes from protobuf and then used to publish acknowledgement.
Is there a chance this may be caused by NATS or NATS Streaming server?

@aricart

This comment has been minimized.

Copy link
Member

commented Jul 17, 2019

@nazar-pc the version for the nats-streaming-server doesn't seem to be correct. The currently released version is v0.15.1. Also it would be interesting to see if the connection_lost handler is getting called. Your application should restart if so. See https://github.com/nats-io/stan.js#connection-status

@nazar-pc

This comment has been minimized.

Copy link

commented Jul 17, 2019

It says version: 2.0.0 at http://localhost:8222/varz, but yes, turns out we have :latest image from Docker Hub currently running.

Yes, my application does have connection_lost callback and does graceful shutdown/restart when that happens. Apparently in this case it was not called for some reason.

This situation also correlates with increased load on our setup, so maybe there is actually something that should fire connection_lost or something like that.

@aricart

This comment has been minimized.

Copy link
Member

commented Jul 17, 2019

@nazar-pc the /varz endpoint is for the embedded nats-server. For nats-streaming you want to hit:
http://localhost:8222/streaming/endpoint
Where endpoint is:

  • serverz
  • storez
  • clientsz
  • channelsz

Wondering if you have a mix of streaming server versions? The libraries are backwards compatible, but if you have a mix of servers, it is possible that the connection_lost functionality is not available there.

@aricart

This comment has been minimized.

Copy link
Member

commented Jul 17, 2019

@nazar-pc Also you wouldn't by any chance be passing or accessing the NATS connection in the client directly. It would be awesome to print a stack trace on the error handler.

@nazar-pc

This comment has been minimized.

Copy link

commented Jul 17, 2019

Hm... we've recently upgraded NATS Streaming, but not NATS itself. Only today we've tried to upgrade to NATS 2.0.2, but had issues and reverted back to 1.4.1. All instances are deployed using k8s operator, so they should all be exactly the same version in cluster.

In my case stack trace looks like this:

NatsError: Subject must be supplied
    at Client.publish (/code/node_modules/nats/lib/nats.js:1541:20)
    at Message.ack (/code/node_modules/node-nats-streaming/lib/stan.js:981:28)
    at QueueClient.<anonymous> (/code/dist/lib/QueueClient.js:99:21)
    at Generator.next (<anonymous>)
    at fulfilled (/code/dist/lib/QueueClient.js:4:58)
    at processTicksAndRejections (internal/process/task_queues.js:85:5)
@nazar-pc

This comment has been minimized.

Copy link

commented Jul 17, 2019

Or do you mean that NATS Streaming has something of v2.0.0 inside and may not play nicely with 1.4.1? I'm not familiar enough with NATS and didn't deploy it myself.

@aricart

This comment has been minimized.

Copy link
Member

commented Jul 17, 2019

My comment was regarding the connection_lost. Only 0.10.0 and newer versions of nats-streaming-server implement it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.