"consumer creation response is empty" for Update on consumer using FilterSubjects

[idiazcir]

Hi all!

We use NATS+Jetstream as a message bus for the microservices (Go) in our application. We recently observed a weird state in one of the service consumers, and we are not able to understand why, nor properly reproduce the error.

The microservices (together with NATS server) run inside Docker containers that are managed with docker-compose. It is not unusual for the machine where those run to reboot from time to time. When that happens, the consumers that are on the services are recreated using CreateOrUpdateConsumer. We recently introduced a consumer that was using ' FilterSubjects ' for the first time for multiple subjects instead of a single one (all the other consumers across all microservices use FilterSubject).

After one of those machine restarts, we saw that the CreateOrUpdateConsumer on that consumer started to fail and return the error ErrConsumerCreationResponseEmpty nats: consumer creation response is empty. We did not find any error messages on the NATS server. Restarting the "faulty" microservice did not solve it. The only way to fix it was to tear down and rerun the whole application, including the NATS server with docker compose down/up.

This leads us to think that the error might be related to the Update path of the CreateOrUpdateConsumer rather than the Create path, which worked after the tear-down, and potentially be related to the usage of FilterSubjects, as it has only happened with the only consumer that use it.

Do you have any idea of what might be happening? Could it be simply some sort of desynchronization due to the reboot? Sorry for not being able to provide more input, as we are not able to reproduce it. For context:

• NATS server version: 2.12.1
• NATS Go client version: 1.47.0
• Consumer config:

	streamConf := jetstream.StreamConfig{
		Name:        "DEVICE_READINGS",
		Description: "Stream for neurona device readings",
		Subjects: []string{
			"neurona.events.devices.reading.>",
			"neurona.events.devices.offline.>",
		},
		Retention:         jetstream.LimitsPolicy,
		MaxMsgsPerSubject: 1,
	}

	consumerConf := jetstream.ConsumerConfig{
		Durable:        "app-devices-consumer-digital-copy-readings",
		FilterSubjects: []string{
			"neurona.events.devices.reading.*.*.reported.v1",
			"neurona.events.devices.offline.*.*.reported.v1",
		},
		DeliverPolicy:  jetstream.DeliverNewPolicy,
		MaxDeliver:     3,
		AckPolicy:      jetstream.AckExplicitPolicy,
		AckWait:        30 * time.Second,
	}

[RenSolvyn]

I could not make your failure happen, and I tried fairly hard with your exact versions and configs, so instead of a guess I can offer you three things: precisely what this error means at the byte level, a set of causes your scenario provably is not, and the one capture that will pin the real cause the next time it fires. Plus one question about your compose file that may reinterpret your only successful fix.

What the error means, exactly. In your nats.go v1.47.0, ErrConsumerCreationResponseEmpty is raised at jetstream/consumer.go:325 only when the server's reply is syntactically valid JSON that contains neither an error object nor consumer info. A malformed reply, a timeout, or no-responders all surface as different errors, so when you see this one, a responder answered with a well-formed but contentless body. Worth knowing: the guard was introduced defensively in Aug 2025 (commit 7af9697, first shipped in v1.45.0), and the PR that added it documents no producing server condition. Your report appears to be novel field data, not an instance of a catalogued bug.

What I ruled out by execution, using nats-server v2.12.1 (your version) and your exact stream and consumer definitions, field for field (the one omission being the stream's Description string, which is metadata): thirteen create/update calls across seven server generations, every one returning full consumer info, zero empty-shaped bodies. The generations covered a fresh store, an idempotent re-upsert, two SIGKILL restarts (the faithful analogue of your machine reboots) with immediate upserts, and four deliberate power-loss corruptions of the consumer's on-disk state before restart: meta.inf truncated to zero, half-written, deleted with the directory kept, and the entire consumer directory emptied. The server self-healed all four and answered correctly. Two conclusions follow. First, a standalone 2.12.1 does not emit this response shape in any restart or corruption state I could construct. One scope note for honesty: my trials are single-node; if your compose actually runs a clustered NATS, say so, because the cluster code paths are the one region these trials do not cover. Second, FilterSubjects is substantially exonerated as the differentiator: the only mechanical difference from your single-filter consumers is that multi-filter consumers use the unfiltered CREATE API subject (v1.47.0 jetstream/consumer.go:300-306); the request and response envelope are otherwise identical, and multi-filter behaved perfectly across every trial.

The question about your compose file: is the JetStream store directory on a named volume? This matters because your one working fix, docker compose down/up, has two readings. If the store lives in the container filesystem or an anonymous volume, down destroys it, and what looked like "restart fixed the update path" was actually "a wiped store made the create path run fresh", which is consistent with everything you observed and moves the suspect entirely to server-side state surviving the reboot. docker compose config on the nats service settles it in one look.

For the next occurrence, three captures that convert this from mystery to diagnosis:

1. Raw bytes. Alongside your CreateOrUpdateConsumer error handler, fire the same request untyped and log the reply: msg, _ := nc.Request("$JS.API.CONSUMER.CREATE.DEVICE_READINGS.app-devices-consumer-digital-copy-readings", reqJSON, 5*time.Second) and print string(msg.Data), where reqJSON is the same {"stream_name":...,"config":{...}} payload. Whatever those bytes are decides everything, including whether the responder was even the JetStream API.
2. nats consumer info DEVICE_READINGS app-devices-consumer-digital-copy-readings --json from the CLI at failure time, plus server logs with -DV around one attempt.
3. A check that nothing else in your ecosystem is subscribed on $JS.API.>; a foreign responder racing the server is one of the few remaining explanations for a well-formed empty body.

Until then, two mitigations. Treat this error as retryable with backoff and an alert, since your data shows it is environmental rather than config-driven. And if you add a fallback that deletes and recreates the durable, do it knowing the precise cost: with DeliverNewPolicy, a recreated consumer starts at messages published after recreation, so anything published between failure and recreation is skipped; if that is unacceptable, capture the stream's last sequence first and recreate with a by-start-sequence policy instead.

If you capture the raw bytes, post them here; that single string will be pinpointable. And given upstream has no documented trigger for this shape, that capture would also make a well-founded nats-server issue.

[RenSolvyn]

Follow-up with substantially stronger exclusion data, since "could not reproduce in thirteen trials" deserved a harder test. I ran a metamorphic campaign against the same standalone nats-server 2.12.1: the invariant under test is the one your error is a violation of (every syntactically valid JSON reply to the consumer API must contain either an error object or consumer info), checked on every single response while the system was stressed along the axes my first pass did not cover, namely concurrency and timing.

The campaign: eight concurrent workers racing create/update, delete, and info on your exact durable and config; the server SIGKILLed every 2.5 to 4 seconds throughout, with a dedicated task hammering upserts from the first accepted connection of each boot to sit inside the JetStream recovery window; then 32 config mutations (overlapping filter subjects, duplicates, a twelve-subject list, an empty-string member, and bogus action values). Result: roughly 31,000 valid-JSON replies checked, about 28,000 successes and 3,000 proper error bodies, and zero invariant violations. Every malformed config produced a well-formed error, never an empty shape. One flagged case turned out to be a bug in my own checker (it demanded a consumer-only field from a stream-create response), which I mention because it shows the harness was actually capable of firing.

What this changes: a standalone single-node 2.12.1 appears essentially unable to produce the response shape you saw, even under kill-storms, boot-window races, and config abuse. That narrows your trigger to the things this setup cannot exercise: clustered JetStream (if your compose runs more than one node), something else in your system answering on $JS.API.>, or environment-specific state. The raw-bytes capture from my previous comment remains the decisive artifact, and this data is exactly why: whatever produced your reply is not the ordinary server path.

[idiazcir]

Thank you very much for the exhaustive response and the depth of the testing to try to reproduce the issue.

Answers to your questions:

• Do we use clusters? We do not use clusters; we have a single NATS server.
• Do we use store data on a binded directory or named volumes? In Docker Compose, we use named volumes. Which, actually, I see now invalidates the premise we had that doing a down caused the consumer to take the "Create" path rather than the "Update" one. This is not correct, as data/volumes were kept intact...

For the mitigations, we have already implemented the second one. We have a fallback with errors.Is(err, jetstream.ErrConsumerCreationResponseEmpty) to delete and recreate the consumer. We will also add the traces you suggested to see if we can obtain more context in case it happens again.

Considering that we weren't able to reproduce the issue, neither you with all the tests conducted, nor is there documentation outlining producers of the server condition; I would not be surprised if the error was system-related or caused by a specific environment state as suggested.

We will keep you updated if we get the same error (this time with traces).
Again, thank you for the answers and the whole investigation conducted. We really appreciate it :)

[RenSolvyn]

Thanks for the thorough follow-up, and glad the fallback is already in place.

The named-volume detail is genuinely useful, and you're right that it rules out the down/recreate theory. With a single server and volumes kept intact across restarts, the Create-vs-Update path premise doesn't hold, so that's one hypothesis we can set aside.

Given that neither of us could reproduce it and there's no documented trigger for the empty-response condition, I think the posture you've landed on is the correct one: the errors.Is(err, jetstream.ErrConsumerCreationResponseEmpty) fallback handles it gracefully regardless of root cause, and the added traces will give us the context we're currently missing if it recurs. For an environment-dependent condition like this, that's more valuable than continuing to speculate without data.

Happy to help dig back in if the traces surface something concrete. Otherwise this seems like a reasonable point to close. Thanks again for the collaboration on it.