A consumer already declares its topics — it hands them to subscribe() on the
next line. Requiring somebody to ALSO list them by hand somewhere else (a
deploy script, a runbook, an infra repo) is a second source of truth, and it
drifted exactly the way second sources of truth do: the ironmemo stand ran
for weeks with six recordings topics missing from its deploy script's list.
The consumers subscribed, got UNKNOWN_TOPIC_OR_PART, logged it at ERROR
several times a second, delivered nothing, and reported healthy the whole
time. That same script was meanwhile creating a topic name nothing had used
since the NestJS port.
The NATS backend never had this failure mode — its stream captures
`<prefix>.>`, so a new topic needs no broker-side change at all. Kafka was
the odd one out; now it creates what it is about to subscribe to, plus each
topic's .dlq (a poison message with no DLQ to park in is a dropped message).
Best-effort by construction: an existing topic is the normal case, and a
broker that refuses creation must never block a consumer that may already
have its topics. KAFKA_PROVISION_TOPICS=false opts out for deployments where
topics are infra-owned and applications hold no create ACL.
UNKNOWN_TOPIC_OR_PART also drops from ERROR-per-poll to one WARNING per
topic. librdkafka re-reports it on every metadata refresh, and at ERROR that
buried every real failure in the same log.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>