AsyncAPI and describing Error/Retry topics

[ivangsa]

I watched the recent retry/DLQ + AsyncAPI session and looked at the labs.

Declaring every retry topic and the DLQ as full channels becomes very verbose (especially with multiple retry levels) and also pushes internal infrastructure into the public API surface.

There’s an open proposal in the AsyncAPI Kafka binding that tries to solve exactly this problem in a more compact way:
asyncapi/bindings#299

Would you be open to supporting something closer to that model (or collaborating on it)?

It seems like it could reduce the verbosity while still giving tools a clear place to declare retry and DLQ topics.

[nashjain]

Thanks @ivangsa this is a very useful perspective, and I agree that declaring every retry topic and DLQ as a full channel can become verbose, particularly when there are multiple retry stages.

My motivation for modelling this at the operation level in the core specification is primarily to keep the semantics protocol agnostic. Retry behaviour and dead-letter routing are not inherently Kafka concepts; similar patterns exist across brokers and transports. The operation should be able to express that processing may be retried and, eventually, routed to a dead-letter destination, regardless of how a particular protocol implements those destinations.

I also think the distinction between “internal infrastructure” and “external API” deserves some nuance. One might think of a retry topic or DLQ as internal to the consuming service, but it is still part of the observable message topology and can affect delivery guarantees, ownership, operations, monitoring, replay, and governance. As a publisher of an event I would certainly want to be aware of these aspects. Hence I feel they should be explicitly documented in the spec and part of the contract.

Perhaps there are two related concerns here:

• The core specification should describe the portable behaviour and relationships: retry policy, retry route, and dead-letter route.
• A protocol binding can describe how those routes are materialized. For example, Kafka topic naming conventions, retry-topic counts, partitions, replicas, retention, and other provisioning details.

In that sense, I see your error-topics proposal as complementary rather than mutually exclusive with the core proposal. The core model could remain concise and protocol agnostic, while the Kafka binding provides an ergonomic way to provision or derive internal retry/DLQ topics without requiring each one to be declared as a separate public-facing channel and operation.

I’d be very interested in exploring whether the two proposals can be aligned around that separation. Looking forward to collaborating with you on this.

[ivangsa]

I agree that retry behavior is important and that it should be documented.

I think it is useful, a distinction: Physical retry topics are not a generic messaging concept. They are an implementation pattern used by log-based streaming platforms such as Kafka and Pulsar. Most other brokers provide retry through broker-managed redelivery and do not require additional physical destinations.

I also wonder if we are attaching the retry topology to the right concept. Retry topics seem to belong to the receive operation, rather than to the published channel. Each consumer can define its own retry policy, so each receive operation can have a different retry topology:

                    orders.created
                         │
          ┌──────────────┼──────────────┐
          ▼              ▼              ▼
      billing         shipping        analytics
      3 retries       no retry        5 retries
      + DLQ            + DLQ          retry topics

If retry topics are attached to the channel, we would need a separate set of retry topics for each consumer. Otherwise, consumers would not be able to distinguish which retry topics belong to them. This would make the specification even more verbose as the number of consumers grows.

Given this, would it make more sense to describe retry topics as part of the binding for the receive operation, instead of modeling them as additional top-level channels? That would still make the topology explicit, but it would keep it associated with the consumer that owns it.

I did't think about Pulsar, but I believe is the other broker that need physical retry topics (aside from Kafka)

What do you think?