ARTEMIS-550 Add virtual topic support

[graben]

First preview for discussion, still lacking junit tests

[clebertsuconic]

can you check this, @mtaylor .. also in reference to the addressing branch

[graben]

Add some initial tests for better review

[mattrpav]

Has there been mention of the optional parameters: selectorAware, transactedSend and concurrentSend options? These are super useful for improving reliability and scalability when using Virtual Topics.

[clebertsuconic]

I am doing a release soon (this week, was supposed to be last week), so please don't merge this until 1.5.0 is out.

[graben]

@mattrpav: I'm not sure these options can be implemented the way the virtual topic solution is integrated. Artemis works a bit different than ActiveMQ. But maybe someone with a deeper view into core code can answer this.

[mattrpav]

I'll detail out the use cases:

1. selectorAware: This allows an interested subscriber to only receive messages in their queue that match a given selector. This broker-side pre-filtering is an optimization which reduces the on-disk requirements for really busy Virtual Topics.

2. transactedSend: This is a (imho & experience) critical reliability requirement. In use cases where reliability is a critical requirement: when a producer sends a message to the broker, even during non-transacted sessions, the broker should transact the dispatch to the consumer queues before ack'n the producer. This ensures no messages are lost in the event of unplanned outage while a broker is dispatching messages to the consumer queues.

3. concurrentSend: This is a critical performance enhancement for Virtual Topics where there are a large number of consumer queues. The behavior is that the broker processes queue dispatch in parallel to consumer queues in order to improve throughput and lower latency.

[Notes]

i. transactedSend and concurrentSend are often used in conjunction

ii. selectorAware requires some sort of persistence store so selectors are maintained across restarts (specifically useful during maintenance and failover testing)

iii. While outside the scope of this PR, I thought I'd mention it here as it is related-- there has long been a desire to enhance ActiveMQ to support single-message store. That is, for all subscriptions, only store a single copy of a message. This becomes really apparent in use cases with a large number of Virtual Consumers.

iv. transactedSend and concurrentSend are also super handy in composite destinations (aka diverts in Artemis). An enhancement to Artemis to support both would be a big win in closing a critical feature gap b/w ActiveMQ 5 and Artemis.

[graben]

Well, I'm not sure at all but IMHO the actual implementation of PostOffice is transacted by default no way to deactivate. Concurrency seems a bit difficult to me to implement in this PR. Might implemented in general? The selector or filter (Artemis way) feature is on my todo list after the basis is approved.

[mattrpav]

@graben

re: Transacted: if the PostOffice is transacted by default, that's great. The option in 5.x felt more like a "shore-up" to get to transacted. I don't see a compelling reason to support non-transacted broker-side message movement. A note in the migration guide would probably suffice.

re: Concurrency: I see this having value, and agree it should be a separate JIRA/PR.

re: Selector/Filter: that's great that you have it on your radar. You mentioned filter, so I feel obligated to clarify-- this selector is determined by the consumer using the standard JMS selector available in the API. The broker then store these to properly filter messages if the consumer goes offline. Does Artemis' filter support this behavior, or is filtering broker-side configuration?

Thanks!

[andytaylor]

@graben Could you explain why virtual topics need to be added. As far as I was aware in ActiveMQ its just to provide shared consumers and queue like semantics on a topic which Artemis does by default?

[rlodge]

@andytaylor How does Artemis provide the same functionality by default (without configuration for each "virtual topic / queue" being added to the broker's configuration)? This is one of the only (maybe the only) features keeping us from attempting to migrate to Artemis from ActiveMQ. I've looked through the documentation and haven't been able to find a similar feature other than diverts, which seem to need broker configuration and broker restart, which would be a dealbreaker, since our environment requires clients to to be able to dynamically create the queue (not just queue-like) destinations on the fly without broker reconfiguration or restart.

[andytaylor]

@rlodge Thats what I am asking :), Artemis supports JMS 2.0 and shared consumers and implements topic style addressing using queues anyway. I'm asking for an explanation as to what else is actually needed and how best this could be added without introducing the notion of a topic at the broker which currently doesn't exist?

[rlodge]

@andytaylor I'm not sure what you mean by "implements topic style addressing using queues"....

[andytaylor]

@rlodge In the core Artemis Broker there is no concept of Topics, they don't exist. Its just a facade on the client and a small amount of config and management. The core Broker only understands Addresses and queues, so when a 'JMS topic' is created nothing actually exists. In reality when the client connects it creates a queue that acts as the subscription. That being the case adding topic functionality to the core might be the wrong approach, So I am trying to understand what tthe actual use case is and how it might fit better into the current core broker concept.

[clebertsuconic]

@rlodge you can have as many consumers you like on a QueueSubscription with Artemis.

You can limite the client-id if you like, but you can also bypass that.

The only use case I can see for artemis with Virtual topics is migration case... users migrating to artemis without changing code.. but I want to make sure I understand the requirements. What is needed? Route the virtual queue to another existing core queue? if that's the case it's pretty simple implementation. (I guess you would have most of what is needed on this PR)?

[mtaylor]

Firstly we are making some major improvements to the Artemis addressing model that might help here.

@mattrpav

selectorAware: This allows an interested subscriber to only receive messages in their queue that match a given selector. This broker-side pre-filtering is an optimization which reduces the on-disk requirements for really busy Virtual

This is how normal Artemis subscriptions currently work. The Artemis addressing/routing model is different from ActiveMQ. Messages are routed straight to subscription queues, and filtering happens on the broker at routing (send) time.

2. transactedSend: This is a (imho & experience) critical reliability requirement. In use cases where reliability is a critical requirement: when a producer sends a message to the broker, even during non-transacted sessions, the broker should transact the dispatch to the consumer queues before ack'n the producer. This ensures no messages are lost in the event of unplanned outage while a broker is dispatching messages to the consumer queues.

We also already have this behaviour, but it works slightly differently. We don't create a transaction that puts the message on all consumer queues then returns an ack. Instead, we persist the message and return an ack (depending on the protocol behaviour). In the event of the broker going down just after the message is persisted, but before routing, it will retry routing on restart. Which gives you the same behaviour.

3. concurrentSend: This is a critical performance enhancement for Virtual Topics where there are a large number of consumer queues. The behavior is that the broker processes queue dispatch in parallel to consumer queues in order to improve throughput and lower latency.

I don't think I fully follow what you are saying here, but, in Artemis there's only consumer queues. We route messages direct to consumer queues. All consumption of messages form those queues and dispatching to the consumer is done in parrallel using a shared thread pool.

iii. While outside the scope of this PR, I thought I'd mention it here as it is related-- there has long been a desire to enhance ActiveMQ to support single-message store. That is, for all subscriptions, only store a single copy of a message. This becomes really apparent in use cases with a large number of Virtual Consumers.

We already do this. A message is only persisted once, we then use reference counting to delete it once it's been consumed from all queues.

[clebertsuconic]

IMHO we should move this discussion to the dev-list, with the following:

i - use cases
ii - required changes.
iii - how it's being implemented on the PR.

when I read these changes I don't have a clue if the 3 points here are being covered, and what performance implications (or even stability) to other regular queue cases.

[mtaylor]

With regards to the general Virtual Topics use case. We're making some improvements to the Artemis addressing model. See https://issues.apache.org/jira/browse/ARTEMIS-780. I think these changes might make it easier to use equivilent "shared topic" behaviour.

It will depend on the protocol you are using as to how this would behave. But, you should be able to either get direct access to a pre-configured queue and add another consumer on it. The abilty to reference queues directly is something we'll be exposing.

Alternatively, you'll be able to subscribe to the same "durable" subscription as another client, by using the same client id (again protocol specific here). For AMQP this would be the container-id, for JMS and STOMP client id.

[mtaylor]

@clebertsuconic +1

@mattrpav and @rlodge if you could provide us some more information about your use cases, we can see if current features or those coming in ARTEIMS-780 can help out. If not we can try to address them.

[rlodge]

A discussion on the dev list would at the very least leave those of us who don't subscribe out of the discussion . . .

Our use case for virtual topics in ActiveMQ is fairly straightforward: we want to be able to broadcast a message to an unknown number of different subscribers without doing something like camel multiplex on the application end. The consumers must be able to do concurrent processing (multiple consumers) with queue semantics. We need the consumers to be able to create and delete their own subscriptions without restarting the broker or doing any application configuration on our end; the consumers must be able to go offline and come back and find their messages waiting for them; waiting messages can't expire or be thrown away. This probably has to be done with the latest ActiveMQ client library and without code changes.

[mattrpav]

@andytaylor its a couple things beyond just shard topic subscriptions (JMS 2.0). #3 being super critical and highly valuable.

1. Using queues instead of durable topic subscribers allows for all the add'l queue features-- exclusive consumers, message groups, etc.
2. It separates the producer and the consumers into separate destinations in an automatic way, without requiring server-side configuration
3. Adding timestamp to print-data #2 allows you to separate producing message flow from consuming message flow between brokers across network connectors. We implement both scenarios all the time: replicate topic production across brokers, but separate consumers.. and the inverse.. share consumers across brokers, but not producers.

Competitor Note: IBM MQ supports the topic -> queue separation. The advantage ActiveMQ's Virtual Topics have is that it doesn't require server-side configuration for every topic -> queue setup and every new subscriber.

[andytaylor]

@rlodge so I think all that is doable without having to add the notion of a virtual topic. With the addressing changes this should be even easier to add to the Openwire protocol manager.

one thing you could explain is does "we want to be able to broadcast a message to an unknown number of different subscribers" mean to different addresses or just one address like myTopic?

[andytaylor]

@mattrpav thanks for that, for point 1, everything supported on a queue is supported on a subscription as thats actually what it is. Maybe Artemis is just not exposing some of these. Again with the new addressing changes this will be easy. Im not sure 3 is applicable since Artemis routes to addresses and consumes from queues. 3 I am not sure I understand.

[rlodge]

@andytaylor We have a single topic, like "VirtualTopic.RecordUpdateNotifications". The system doesn't have a list of who's receiving it; downstream systems subscribe to Queues with the right name pattern and receive the updates. To them, it's like any other Queue. The upstream system and the broker configuration aren't modified when a new consumer comes on line. The key is no server or application configuration to make it work, and also all queue features (especially message groups) being available.

[andytaylor]

@rlodge cheers, makes sense. All that is implemented now, It just needs exposing in the protocol layers when an Openwire JMS client connects to a virtual topic. Like I mentioned before the addressing changes will make it easier to expose this as well.

[mtaylor]

@rlodge

So one thing we are working on right now is adding is the ability to directly address a queue from a consumer. All of this is upcoming in Artemis-780 jIRA (which I will post more info on very soon).

But the key points are:

1. You'll be able to specify routing semantics on addresses. e.g.

Essentially what this means is that any queue created with an address "myTopic" will get all messages sent to myTopic. To create a subscription all we do internally in Artemis is create a new queue with address "myTopic".

To do Point to point or "queue" sematnics, you simply configure the address to have routing type anycast.

In Artemis if a consumer requests consuming from myQueue we look to see if a queue already exists, if so we add a consumer, other wise we create one and add a consumer.

As part of the work we'll also allow clients to specically reference a queue directly. Using the pattern:

address::queueName

This means I want to consume from a specific queue called, "queueName" with address "address". If the queue doesn't exist it will create it (providing you have autoCreateQueues enabled). This allows you to have several clients, creating subscriptions to the same queue.

Whilst ARTEMIS-780 is meant to cover the changes to the CORE addressing model, to enable all this extra stuff, there's also additional tasks to offer this support over the various protocols. To get ActiveMQ JMS 1.1 clients working, we'll also need to update the OpenWire protocol.

[rlodge]

@mtaylor What I'm hearing is that basically, following changes in 780, a client would be able to send to an address "RecordUpdateNotifications", other clients could subscribe to "RecordUpdateNotifications::SystemARecordUpdateNotifications" and it would act essentially like virtual topics do now; all "RecordUpdateNotifications::*" subscriptions would read from queues coming from the single address?

If so, do OpenWire clients need to use a specific queue / address naming scheme, or would it happen automatically somehow (and could it be mapped from the VirtualTopic.* naming scheme)? It's not a terribly large problem for us to go through and change our queues from VirtualTopic.* to a slightly different naming scheme, though we'd rather change nothing about the clients; it would be more difficult to somehow change the API's we were calling...

[graben]

@andytaylor: One big advantage of queue semantics vs topic subscriptions is the usage of shared pooling. No need for clientId@JMSComnection. Every single pool connection can be used for every topic (sending) or every queue (receiving or sending). Maximum flexibility. As far as I know this is still not addressed in JMS 2.0 API.

[andytaylor]

@graben Im not sure what you mean by shared pooling but if you mean a pool of consumers subscribed to the same topic then thats what JMS 2.0 gives you. Maybe I'm misunderstanding what you mean by pooled connections.

[graben]

@andytaylor: I mean the usage of one single pool which is used to send/receive concurrently to multiple topics and virtual topic consumers. Such are often used in integration infrastructures like e.g. Apache ServiceMix. You only have to configure one pool which addresses every purpose since you do not have to register clientId to the jms connections when consuming from queues only. I think that's different to pooled subscribed users in JMS 2.0.

[andytaylor]

@graben It sounds different, what is it a pool of, connections?

[mattrpav]

@graben agreed.

Queue Consumer use cases:

Connection(s) | # Queue(s) | Option | Desired Behavior

---------------- | ----------- | ------- | ------------------
1 | 1 | | Single threaded, ordered message consumption
1 | 1..n | | Single threaded, ordered consumption of multiple queues (comma separated today)
1..x | 1..y | Using connection pool | multithreaded, ordered consumption using shared connection pool (same clientId). Each thread getting ordered messages for their specific queues.
2 | 1 | | Round-robin dispatch of message consumption
2 | 1 | exclusiveConsumer | Single threaded, order consumption w/ HA standby consumer
2 | 1 | exclusiveConsumer+JMSXGroupId | Multi-threaded, order consumption w/ active-active consumers separated by header value

[graben]

@andytaylor: Have a look into http://activemq.apache.org/how-do-i-use-jms-efficiently.html

[dbit-xia]

AMQP protocol does not seem to support virtual topic !

[jbertram]

@dbit-xia, this PR is closed. If you have an issue you want to discuss please use the ActiveMQ User mailing list and provide a full explanation of the problem.