Support "fire-and-forget" messages

[joseph-wakeling-sociomantic]

For some use-cases it is not vital that every message gets through, but it is important to avoid queue overflow in the case that there is not an active channel reader. Some example use-cases:

• an app wants to start writing a new DMQ channel, but doesn't want to block on the reader app being implemented

  • this allows writers to get their work done (and test potential performance issues) without worrying about the reader
  • can switch to persistent messages if needed only once the reader is ready

• we have a very high-throughput channel where if the reader goes down, we will overspill very quickly due to the quantity of data

  • if we can tolerate the data loss we might want to just drop messages rather than create a big backlog and overspill onto disk

• we have a channel where the reader needs to focus on the most current data, and where old messages can be discarded

  • in this case, processing a DMQ backlog when the reader revives might delay restoration of a well functioning system (even if the size of the backlog is tolerable)

In all of these cases it would be convenient to have either a request type (or a flag for Put requests) that indicates that, in the absence of a channel reader, the DMQ node can just drop the message.

[joseph-wakeling-sociomantic]

An extra compromise option might be for messages to be allowed a specified timeout period (in seconds) in which a reader can collect them, before they are dropped.

This might allow us to have a slightly more flexible approach that avoids data loss due to short term network issues but starts dropping messages in the case of longer-term outages. It could be tailored on a use-case-by-use-case basis.

[david-eckardt-sociomantic]

Since the reader needs to focus on the most current data, the least current records need to be dropped. Here is my suggestion: If a fire-and-forget message is pushed to a DMQ channel and it doesn't fit in the memory queue, pop and drop messages from the memory queue until there is enough space to push the new message. In addition, if there is a disk overflow, drop that, too, because it always contains even older messages.

[gavin-norman-sociomantic]

Here is my suggestion: If a fire-and-forget message is pushed to a DMQ channel and it doesn't fit in the memory queue, pop and drop messages from the memory queue until there is enough space to push the new message. In addition, if there is a disk overflow, drop that, too, because it always contains even older messages.

That's exactly what I was going to suggest as well 👍