“ERROR: Subject remapping requires Options.oldRequestStyle() …”

[gdiet]

In our cloud production system, we use java NATS 2.8.0 client together with docker nats:2.1.9-linux NATS server (no JetStream involved). Occasionally (a couple of times per week) the NATS client prints to stdout:

ERROR: Subject remapping requires Options.oldRequestStyle() to be set on the Connection

This error message was introduced with the "Subject Remapping Fix" in #327

We don't know the exact circumstances that trigger the message. Logging the whole NATS traffic (for investigations) would be possible but cumbersome.

I see the following issues:

• There seems to be no documentation available when this message can occur and whether / which problems might arise if I simply ignore the message.
• This is the only use of System.out in the Java NATS client except for the connection tracing that can be enabled as option, see io.nats.client.Options#isTraceConnection. Is it intended that System.out is used for signalling potential problems?
• The javadoc of Builder.oldRequestStyle() says "Turn on the old request style that uses a new inbox and subscriber for each request." This sounds to me like the new request style (which probably re-uses inbox and subscriber) in general is regarded as "better", and you use the old request style as "last resort". What I'm missing here is documentation helping me to decide.

[scottf]

@gdiet Do you re-use / re-send message objects? We are able to reproduce this under those conditions. I had also seen it intermittently but couldn't quite put my finger on it.
As it turns out, the error message is a misleading anyway. It happens when we can't match the incoming message with the outgoing one / future that made the request. This should actually never happen. Since it was out-of-band (async) there really wasn't much we could do so printing a messages was better than nothing.
The good news is the current 2.9.0-SNAPSHOT has a fix regarding message handling that addresses this problem. Can you try upgrading to 2.9.0-SNAPSHOT and see if this fixes your problem? 2.9.0-SNAPSHOT is stable and backward compatible since it is almost exclusive JetStream additions and improvements.

[gdiet]

@scottf We call message.getReplyTo() multiple times to send multiple reply messages. (For this, the original messages have been sent with publish, not with request.)
Yes, I could try using 2.9.0-SNAPSHOT - however, I can't find it published on maven central. Am I overlooking something?

[scottf]

@gdiet It's up on sonatype in the snapshot repo You should just need to change your version number

<dependency>
  <groupId>io.nats</groupId>
  <artifactId>jnats</artifactId>
  <version>2.9.0-SNAPSHOT</version>
</dependency>

You might need to add sonatype to your repos

https://oss.sonatype.org/content/repositories/snapshots/

[gdiet]

@scottf Thanks, that did the trick. I'll get the snapshot version deployed and will check in the next few days whether or not I still get these Subject remapping messages.

[gdiet]

@scottf with 2.9.0-SNAPSHOT, up until now the message didn't appear again. I hope it doesn't reappear over the weekend - it often pops up on Saturday during system maintenance. I'll post an update on Monday. Thanks to the team for a great MQ server & Java library!

[gdiet]

@scottf No more occurrences of the "Subject remapping" message over the weekend. Yay! I close the issue.

[Zannith]

Have swapped out my pub/sub mechanism with nats 2.10.0 on Java 11 and have run into this issue. Will be gone for a bit, but when I am back I will try to make a small reproduceable example.
ERROR: Subject remapping requires Options.oldRequestStyle() to be set on the Connection _INBOX.eRtA1WbozXhRrlfB2s055U.eRtA1WbozXhRrlfB2s05Pj eRtA1WbozXhRrlfB2s05Pj

Here is another error i'm getting in case it is related
consumer deliver subject forms a cycle (500)

[scottf]

@Zannith Please send more information when you get a chance.

[Zannith]

The cause of this error appears to be having having all subjects on a stream with the > wildcard. Maybe this is expected behavior? But I think it could still have a better error. Changing the subject of line var dataStream = StreamConfiguration.builder().name("Live-Data").addSubjects(">").storageType(StorageType.Memory).build(); to "Live-Data.>" removes the error from this example

Here is my example test

public class DatabusTests {
	private static Connection nc;
	private static JetStream js;
	private ConsumerConfiguration defaultConsumerOptions = ConsumerConfiguration.builder().deliverPolicy(DeliverPolicy.Last).build();
	private PushSubscribeOptions defaultPushSubscribeOptions = PushSubscribeOptions.builder().stream("Live-Data").configuration(defaultConsumerOptions).build();
	
	@BeforeAll public static void setup() throws IOException, JetStreamApiException, InterruptedException {
		try {
			nc = Nats.connect(new Options.Builder().server("nats://0.0.0.0:4222").build());
			if (!nc.jetStreamManagement().getStreamNames().contains("Live-Data")) {
				var dataStream = StreamConfiguration.builder().name("Live-Data").addSubjects(">").storageType(StorageType.Memory).build();
				nc.jetStreamManagement().addStream(dataStream);
			}
			js = nc.jetStream();
		} catch(Exception ex) {
			fail(ex);
		}
	}

	@AfterAll public static void tearDown() throws IOException, InterruptedException, JetStreamApiException {
		nc.jetStreamManagement().deleteStream("Live-Data");
	}
	
	@Test
	public void testErrors() {
		try {
			AtomicBoolean messageReceived = new AtomicBoolean(false);
			MessageHandler handler = (msg -> {
				messageReceived.set(true);
			});
			var disp = nc.createDispatcher(handler);
			js.publish("Live-Data.Topic1.subtopic", "data".getBytes());
			js.subscribe("Live-Data.Topic1.subtopic", disp, handler, true, defaultPushSubscribeOptions);
			Thread.sleep(100);
			assertTrue(messageReceived.get());
		} catch(Exception ex) {
			fail(ex);
		}
	}
}

[ripienaar]

You cannot have a stream with > as a subject at all. Client should catch and error on that if not already.

[Zannith]

@ripienaar Okay, so it only supports *? The above test is actually working with the > though, so long as I have something before it. And I am getting no error with it. Is that something I should file a new bug for?

[ripienaar]

It supports wildcards, but what > means is all subjects including internal subjects, apis etc, it will break a lot of things. You can only make a stream with > as a subject if you disable acks - and that would then be very minimal applicability. Best to design a subject space for your apps and persist just what needs it.

[Zannith]

Ah okay that makes sense, So I would need to be certain that whatever subject prefix I have does not crossover into any of the internal subjects. Thank you for the clarification!

[scottf]

@Zannith You cannot set a stream subject to have a wildcard, but your subscribe subject can have a wildcard, I'll add validation for this. See Issue #447

As far as your test, I think you need to subscribe before publishing. The handler in the dispatcher can be null or you can use the nc.CreateDispatcher() no param method, but the handler is not in use until the subscribe since the dispatcher does not yet know which subject to send to it.