When is NATSNoRespondersException thrown other than when the Stream/Subject is not existing ?

[andronachev]

Hello,

TL;DR: we get NATSNoRespondersException on the productive environment, but the Stream,Subject and Consumers (subscriptions) are properly setup so question is: when/why would this exception be raised for other reasons ? Not reproducible locally unless Stream does not exist.

We are facing an issue only on the prodcutive environment and I am not able to reproduce it. We get thousands of NATSNoRespondersException on Publishing a message to jetstream.

Exception is thrown when calling IJetStream. Task PublishAsync(string subject, byte[] data); method.

So far we are handling this exception on purpose, in order to upsert the Stream and Subject on the server, to achieve kind of a "self configuring at runtime" of our publishers and subscribers on the NATS JetStream.

However this spamming of NATSNoRespondersException is happenign after the Stream exists, the Subject exists, so something else is causing this exception adn I have no idea what:

• is it a lack of consumers defined on the server for the Subject ? no, tested that on localhost and anyway there is the consumer on the server in prod, I checked
• too many concurrent publishes ? not sure, but on localhost I tested with 3 nested parallel foreaches of 1000, 37 and 25 items respectively so we can do the math how many publish and subscribe that is to the same stream :D : result is: no problem

So what other scenarios may be causing teh NATSNoRespondersException ?

No logs or error logs of any kind on the JetStream docker pod:

Thoughts ?

[andronachev]

What I don't understand, is that I see in this pull request when this exception was introduced in NATS.NET, it says that it's thrown when "there are no subscribes" : #413

However, I am testing locally with a single stream and 600 subjects to it, and thousands of parallel publishes on these subjects and the stream has zero consumers on the server (durable or ephemeral, no type of consumer at all). Did not call subscribe on this stream or any subject. And there is no exception. The messages are stored on the nats server (the stream is configured with File retention... until delivery)

[scottf]

So there are 2 cases where you get no responders.

1. When you make a core request and there are no repliers listening to the subject.
2. When you JetStream publish to a subject that does not exist. Consider that JS Publish is actually a core request where the server/subject is the replier.

You said the subject exists, but this can't be the case.

[andronachev]

Hi @scottf , thank you for answering.

Can you detail a bit more on point 1 ? : "When you make a core request and there are no repliers listening to the subject." :

1. what does "core request" refer to: it means when we publish a message, right ?
2. "repliers listening to the subject" ? - is this referring to consumers (durable or ephemerals?) because i mentioned I tested on localhost, and with no Consumers, it does not throw this exception, so I assume "repliers" referrs to something else

About point 2: is clear. the Stream and Subject are there

[scottf]

1. Core request is a non-JetStream base (core) nats request. A replier would be a core subscriber. There are no JetStream streams involved.
2. For a JetStream publish, the server is the replier. If it finds the subject, it stores it based on the stream configuration and "replies" with a Publish Ack

[scottf]

The client is just elevating via exception when the server gives no-responder.

[scottf]

In your post you said

So far we are handling this exception on purpose, in order to upsert the Stream and Subject on the server, to achieve kind of a "self configuring at runtime" of our publishers and subscribers on the NATS JetStream.

Can you please be specific about what you mean by upsert? Are you continually modifying the stream for subjects? I'm guessing, but I'd expect that there might be a small window where subjects are not available during a stream configuration change.

Have you considered using a wildcard subject instead. Something like "foo.*" and you could publish and consumer to "foo." You would never have to update the stream config for subject changes.

[andronachev]

So since we are using only publish from the IJetStream object I am guessing we deffinetely not doing "core requests".

I agree there may be a small window/delay till the Stream is updated. But this "real-time upsertion" of Streams is not meant to be day-to-day, it's meant to happen let's say at moment T=0 when all our micro-services start-up for the first time, and they do their first Subscription. So even if this "hickup" would happen, I would have expected 1-2 exceptions. and those being 2 days ago when we started the services. But we are getting spammed with thousands of such exceptions per 15 minutes, so it's consistent

I did not use wildcard subjects to be honest, because I am not sure about pros and cons of using a single wildcard subject or multiple "fixed" subjects.

But now, we are getting the NATSNoRespondersException at Publish time and the Streams and Subjects exist already (I am checking with nats cli directly on the server). So from the last statement:

"For a JetStream publish, the server is the replier. If it finds the subject, it stores it based on the stream configuration and "replies" with a Publish Ack"

1. The Server is there, online
2. The Stream exists, the Subject exists

Would it be possible that this is in fact some kind of timeout on the Server-side ?

[scottf]

Could you post your code that creates your stream and does your publish? As far as wildcards, I think it's just better to not keep changing the stream config, and instead create it outside of your application code as an administrative function. The wildcards work well, they've been around forever. You can deal with them administratively as well.

[andronachev]

Hi, yes, here is a simplified version that is basically what we do. may have some mistakes because I quickly put it together:
The thing is, as you can see we do a recursive call in the Publish method in the Catch, so my assumption is, at some point, the message does go through, otherwise my pod would be down 100% of the time with out of memory exception. At the moment it does restart often but not down completely

public IConnection GetNatsJetStreamConnection(ConnectionOptions options)
{
	Options connectionOptions = ConnectionFactory.GetDefaultOptions();
	connectionOptions.Url = options.Url;
	connectionOptions.AllowReconnect = true;
	connectionOptions.Name = $"{options.ClientId}-{Guid.NewGuid()}";
	ConnectionFactory connectionFactory = new ConnectionFactory();
	var connection = connectionFactory.CreateConnection(connectionOptions);
	return connection;
}
		
public async Task Publish<TData>(string subject, TData payload) where TData : class
{
	var connection = GetNatsJetStreamConnection(_options.Value);
	try
	{
		var serializedData = JsonConvert.SerializeObject(payload);
		var context = connection.CreateJetStreamContext();
		await context.PublishAsync(subject, Encoding.UTF8.GetBytes(serializedData));
	}
	catch (NATSNoRespondersException ex)
	{
		_logger.LogException(ex, LogLevel.Error, $"ATSNoRespondersException when trying to publish subject {subject}");
		JetStreamUtils.CreateStream(connection.CreateJetStreamManagementContext(),
			NatsJetStreamConstants.StreamName,
			subject);
		await Publish(subject, payload);
	}
}

private StreamInfo CreateStream(IJetStreamManagement jetStreamManager, string streamName, params string[] subjects)
{
	StreamConfiguration streamConfiguration = StreamConfiguration.Builder()
		.WithName(streamName)
		.WithStorageType(StorageType.File)
		.WithMaxBytes(1503238554) //1.4GB max size of the Stream. server is configured to max 2 GB
		.WithMaxAge(Duration.OfDays(1)) //delete messages after 1 day
		.WithDiscardPolicy(DiscardPolicy.Old) 
		.WithRetentionPolicy(RetentionPolicy.Limits)										 
		.WithSubjects(subjects)
		.Build();

	streamInfo = jetStreamManager.AddStream(streamConfiguration);
	return streamInfo;
} 

[scottf]

You stream configuration is mostly fine. Like I said before, make the stream once, use the subject wildcards as I described before, that's what they are designed for, and leave the stream alone. Constantly modifying a stream configuration is definitely a bad idea.

You are creating a new connection for each publish. (CreateConnection is not a pool, it's a full, new connection). This is a very expensive operation and will cause terrible performance if you are publishing a lot of messages. Maybe if you publish once every 10 minutes, fine. If you are publishing quite a bit, this is probably the cause of the problem, the server has so much work to do, it can't respond to other work and No Responders may simply be the symptom of this. Also see note below about publishing as a possible cause.

I would make the context once per connection. and pass it to the publisher. The context itself isn't expensive, it's just unnecessary to continually make a new one and is thread-safe.

I would get rid of the recursion. This construct to me seems like it is asking for trouble. Handle the failure of the publish in the place that calls the publish.

Also about publishing. You can't just slam the server with an unlimited amount of async publishes, the server can't handle it. You need to collect these tasks in batches, maybe 50 or 100, and don't publish more until those publishes have all acked. (You can also have 2 threads, one publishing, queue the task and let another thread check the ack, but you need to build care into that to handle errors and also to not have too many outstanding, maybe a limited size queue) In looking for an example I realize this needs improvement (https://github.com/nats-io/nats.net/blob/master/src/Samples/JetStreamPublishAsync/JetStreamPublishAsync.cs)

[andronachev]

In our 'real code' the IConnection is a singleton, so that's not an issue, but thanks for the details it surely only can help us understand better... the truth is I find the NATS / JetStream documentation quite poor, to be polite.

You mentioned about "unlimited amount of async publishes". Is there a solution to this out of the box ? like in the old nats-streaming with the STAN.NET Client, there was the possibility of setting "MaxPubAcksInFlight" (https://docs.nats.io/legacy/stan/intro) on the ConnectionOptions (I am not familiar with the details, I see this in our own git history... some team members from the past did this.

I assume this is exactly the reason they used this configuration, to throttle the amount of publishes

[andronachev]

also, the point makes sense in principle, however I don't think this is the case now, becasue I ran a nats con report and the result is:

The rest of the report row are ~450 durable consumers all with the same values (zeroes or default)
So I have 34 Akc pending and 11 Redelivered messages, on that consumer and that's it

And I've also been stress testing a local nats jetstream server instance, trying to reproduce the error but with no luck, and I published 1000 x 35 x 25 publish all in parallel, and they all went through