Are fanout supported?

[robfrank]

Are fanout queues supported?
We are heavy users of kestrel and this project looks very interesting, but we use a lot of fanout queues.

[bogdanovich]

Fanout queues are not supported now, but it's fairly easy to add this feature.
However, I'm working on "multiple consumer groups" feature and it's almost done.

You can check v0.5 branch. The idea is to be able to read any queue multiple times.
Usage: get <queue>:<consumer_group>
get work:my_consumer_group
get work:1

It will create a consumer group or durable cusor so you can read the source queue multiple times. It's lightweight and doesn't multiply disk space usage.

What is your fanout queues use case?
Is it really complex? or you can get benefit from multiple consumer groups feature?

[robfrank]

We use fanouts for different purpose; given a base queue named messages:

• test: fetch messages with a test/pre-production sw messages+TEST
• backup: messages+WEEKLY_BACKUP
• fetch with multiple sw component same data: messages+product_1, messages+product_2

We are a java-shop and we use memcached client to access Kestrel servers. We have some clients written in c#.
If you implement fanouts, siberite could be a replacement for kestrel as is. If consumer group is only a suffix for the queue's name, it's quite the same. I don't understand how can you consider expired an item in the queue, because you can't know how many clients are part of the group.

[bogdanovich]

I don't understand how can you consider expired an item in the queue, because you can't know how many clients are part of the group.

1. When you read an item in a usual way: get messages, an item gets expired and deleted.
2. When you read an item using consumer group: get messages:cgroup, a durable cursor cgroup gets initialized and with every read it shifts forward without deleting any messages in the source queue. So the source queue remains untouched. Number of consumer groups per queue is not limited.
3. You can start reading from the source queue directly again and it will start deleting messages. Any existing consumer group cursor that internally points to already deleted message will catch up and will start serving messages from the current source queue head.

Maybe consumer group term is confusing and I should start using durable cursor instead.

PS. I will add fanout queues support to the roadmap.
The implementation is pretty much straightforward.

[robfrank]

Ok, fanout are different. Yes, open a fanout will double the space, but it allows to fetch from main o fanouts independently.
Thank you very much for your support.`

[bogdanovich]

They are supported now.