How to configure connection router

[eanbiso]

Hi team,
I would need to understand if it is possible to configure (and how) a connection router (see connection-router doc) in the k8s context via the Artemis CRD.
Is there some documentation I can look into?
I would need to configure a connection router to redirect MQTT connections with a given clientId always to the same node when we have a deploy with size >=2 (e.g. spec.deploymentPlan.size=2).
Many thanks in advance,

Andrea

[brusdev]

Hi @eanbiso, you can use broker properties, i.e. https://github.com/artemiscloud/activemq-artemis-operator/blob/v1.0.16/controllers/activemqartemis_controller_test.go#L2599

For further details on broker properties see https://activemq.apache.org/components/artemis/documentation/latest/configuration-index.html#broker-properties

[eanbiso]

Hi @brusdev, many thanks for your reply. I will look into that and do some tests.

[eanbiso]

Hi @brusdev, I was trying the proposed connection router configuration but I would need a couple of clarifications about the linked example code:

1. Can you explain to me which is the goal of the line "addressesSettings.\"LB.#\".defaultAddressRoutingType=ANYCAST", // b/c cannot set linkTargetCapabilities from amqp client and if it is always required?
2. Can you tell me how the config "connectionRouters.autoShard.localTargetFilter=NULL|${STATEFUL_SET_ORDINAL}|-${STATEFUL_SET_ORDINAL}",
   works? What would happen if the local target filter is not specified?
3. I would like to enable the cache on the connection router... following your example, should I do that adding the following two properties?

"connectionRouters.autoShard.cache.persisted=true"
"connectionRouters.autoShard.cache.timeout=60000"

Many thanks in advance for you time.

[brusdev]

Hi @eanbiso, those questions are more related to the Apache ActiveMQ Artemis project than the ArtemisCloud operator, I have answered briefly below and for further details you can send an email to users@activemq.apache.org

1. Can you explain to me which is the goal of the line "addressesSettings.\"LB.#\".defaultAddressRoutingType=ANYCAST", // b/c cannot set linkTargetCapabilities from amqp client and if it is always required?

the defaultAddressRoutingType setting is not required with MQTT clients, it is useful only for clients that support multiple routing types like AMQP

2. Can you tell me how the config "connectionRouters.autoShard.localTargetFilter=NULL|${STATEFUL_SET_ORDINAL}|-${STATEFUL_SET_ORDINAL}",

the localTargetFilter allows to define a filter for incoming client connections that will be accepted even if they don't match policy. In that example the CONSISTENT_HASH_MODULO policy transforms client IDs in number modulo 2 so each deployed broker will only accept incoming client connection with a transformed client ID that match its stafulset ordinal.

works? What would happen if the local target filter is not specified?
3. I would like to enable the cache on the connection router... following your example, should I do that adding the following two properties?

the cache is not required with the CONSISTENT_HASH_MODULO policy

[eanbiso]

Hi @brusdev,
many thanks for the provided info.

I've just a latest question, always related to the cache configuration, to better understand how to configure that in the operator context, if I'll need to use a policy different from CONSISTENT_HASH_MODULO.
Since I saw that the ConnectionRouterConfiguration.java has a setCacheConfiguration(CacheConfiguration cacheConfiguration) method, and the CacheConfiguration.java has the setters setPersisted(boolean persisted) and setTimeout(int timeout), I tried to set the broker properties:

"connectionRouters.autoShard.cacheConfiguration.persisted=true"
"connectionRouters.autoShard.cacheConfiguration.timeout=60000"

but on the Artemis CRD I see the error messages:

Message:               some properties resulted in error on pod amq-broker-ss-0 : reasons: map[broker.properties:[{"value":"connectionRouters.autoShard.cacheConfiguration.persisted=true","reason":"Resolved nested property for:cacheConfiguration.persisted, on: Broker Configuration (clustered=true,journalDirectory=/opt/amq-broker/data/journal,bindingsDirectory=/opt/amq-broker/data/bindings,largeMessagesDirectory=/opt/amq-broker/data/large-messages,pagingDirectory=/opt/amq-broker/data/paging) was null"},{"value":"connectionRouters.autoShard.cacheConfiguration.timeout=60000","reason":"Resolved nested property for:cacheConfiguration.timeout, on: Broker Configuration (clustered=true,journalDirectory=/opt/amq-broker/data/journal,bindingsDirectory=/opt/amq-broker/data/bindings,largeMessagesDirectory=/opt/amq-broker/data/large-messages,pagingDirectory=/opt/amq-broker/data/paging) was null"}]]

can you tell me if I did something wrong in the cache configuration?

Many thanks in advance

[brusdev]

Hi @eanbiso, the cache configuration by using broker properties is not supported yet but I don't see any technical obstacles. Feel free to create a new issue for Apache ActiveMQ Artemis or send an email to users@activemq.apache.org.

[eanbiso]

Thanks, I opened a new discussion for tracing this request: Connection Router Cache configuration by broker properties

[brusdev]

Cool, I'll reply on that thread, thanks for starting the discussion.

[eanbiso]

Hi @brusdev, sorry if I bother you again, but I still have a couple of points that I would like to clarify about Connection Router configuration via the Artemis CRD with a deployment plan of size 2 (with the two Broker pods in the same cluster):

1. the definition of a Connection Router PoolConfiguration via the broker properties is not required, right?
2. in the test code you pointed me to, I see the pool local-target-enabled option is not configured (remaining to its default value that should be false)... can you explain to me the reason for this choice?

When you have a bit of time can you give me some insights here?
I would like to be sure to have configured the broker in the right way to reach the goal of spreading the connections among the different pods, based on the hash of their clientId.
Many thanks in advance

[brusdev]

@eanbiso the pool is required only to redirect incoming client connections and the connection router cofigured in the auto sharding test only reject incoming client connections according to the policy CONSISTENT_HASH_MODULO so it doesn't require a pool.
When the local-target-enabled settings of a pool is true a broker adds itself to the pool. In this way, also the broker receiving the incoming client request can accept it.

Could you share your current broker properties related to the connection router?

[eanbiso]

Hi @brusdev ,
first of all thanks for your support.
I will try to better clarify our config scenario and the goals we would like to reach, so that maybe you can suggest us the right configuration to apply.

• Scenario: artemis deployed via operator with deployment size set to 2.
• Goal: routing the incoming MQTT connections on the different nodes, based on the clientId value, i.e. redirect a connection with a clientId "XYZ" always to the same node of the cluster, and in the meantime being able to statistically spread the connections (that use different clientIds) between the different nodes of the cluster
• Configuration: we would like to configure a Connection Router to reach the previous goal

Below the technical details.

1. We have deployed an Artemis CRD with the following config:

  Deployment Plan:
    ...
    Message Migration:    true
    Persistence Enabled:  true
    Size:               2
	...
	
  Spec:
  Acceptors:
    Need Client Auth:  true
    Port:              9000
    Protocols:         MQTT
    Ssl Enabled:       true
    Ssl Secret:        amq-broker-tcp-mqtt-tls-certs-secret
    Verify Host:       false
    Expose:            false
    Name:              tcp-mqtt-tls
	....
 
  Broker Properties:
	...
    connectionRouters.autoShard.keyType=CLIENT_ID
    connectionRouters.autoShard.localTargetFilter=NULL|${STATEFUL_SET_ORDINAL}|-${STATEFUL_SET_ORDINAL}
	connectionRouters.autoShard.policyConfiguration=CONSISTENT_HASH_MODULO
    connectionRouters.autoShard.policyConfiguration.properties.MODULO=2
    acceptorConfigurations.tcp-mqtt-tls.params.router=autoShard
	...

2. Looking into the Artemis amq-broker/etc/broker.xml file of the two pods (amq-broker-ss-0 and amq-broker-ss-1) we see:

• amq-broker-ss-0

	...
    <connectors>
        <!-- Connector used to be announced through cluster connections and notifications -->
        <connector name="artemis">tcp://amq-broker-ss-0.amq-broker-hdls-svc.edge.svc.cluster.local:61616</connector>
    </connectors>
	...
	  <cluster-user>EfdaTVp9</cluster-user>
      <cluster-password>1Rm7U8Ye</cluster-password>

      <broadcast-groups>
		<broadcast-group name="my-broadcast-group">          
			<jgroups-file>jgroups-ping.xml</jgroups-file>          
			<jgroups-channel>activemq_broadcast_channel</jgroups-channel>          
			<connector-ref>artemis</connector-ref>       
		</broadcast-group>	
      </broadcast-groups>

      <discovery-groups>
		<discovery-group name="my-discovery-group">          
			<jgroups-file>jgroups-ping.xml</jgroups-file>          
			<jgroups-channel>activemq_broadcast_channel</jgroups-channel>          
			<refresh-timeout>10000</refresh-timeout>       
		</discovery-group>	
      </discovery-groups>

      <cluster-connections>
		<cluster-connection name="my-cluster">          
			<connector-ref>artemis</connector-ref>          
			<retry-interval>1000</retry-interval>          
			<retry-interval-multiplier>2</retry-interval-multiplier>          
			<max-retry-interval>32000</max-retry-interval>          
			<initial-connect-attempts>20</initial-connect-attempts>          
			<reconnect-attempts>10</reconnect-attempts>          
			<use-duplicate-detection>true</use-duplicate-detection>          
			<message-load-balancing>ON_DEMAND</message-load-balancing>          
			<max-hops>1</max-hops>          
			<discovery-group-ref discovery-group-name="my-discovery-group"/>       
			</cluster-connection>	
      </cluster-connections>
	...

• amq-broker-ss-1

	...
	<connectors>
        <!-- Connector used to be announced through cluster connections and notifications -->
        <connector name="artemis">tcp://amq-broker-ss-1.amq-broker-hdls-svc.edge.svc.cluster.local:61616</connector>
    </connectors>
	....
	  <cluster-user>EfdaTVp9</cluster-user>
      <cluster-password>1Rm7U8Ye</cluster-password>

      <broadcast-groups>
		<broadcast-group name="my-broadcast-group">          
			<jgroups-file>jgroups-ping.xml</jgroups-file>          
			<jgroups-channel>activemq_broadcast_channel</jgroups-channel>          
			<connector-ref>artemis</connector-ref>       
		</broadcast-group>	
      </broadcast-groups>

      <discovery-groups>
		<discovery-group name="my-discovery-group">          
			<jgroups-file>jgroups-ping.xml</jgroups-file>          
			<jgroups-channel>activemq_broadcast_channel</jgroups-channel>          
			<refresh-timeout>10000</refresh-timeout>       
		</discovery-group>	
      </discovery-groups>

      <cluster-connections>
		<cluster-connection name="my-cluster">          
			<connector-ref>artemis</connector-ref>          
			<retry-interval>1000</retry-interval>          
			<retry-interval-multiplier>2</retry-interval-multiplier>          
			<max-retry-interval>32000</max-retry-interval>          
			<initial-connect-attempts>20</initial-connect-attempts>          
			<reconnect-attempts>10</reconnect-attempts>          
			<use-duplicate-detection>true</use-duplicate-detection>          
			<message-load-balancing>ON_DEMAND</message-load-balancing>          
			<max-hops>1</max-hops>          
			<discovery-group-ref discovery-group-name="my-discovery-group"/>       
		</cluster-connection>	
      </cluster-connections>
	...

3. At the broker start up (on one of the two nodes) we can see the log:
   INFO [org.apache.activemq.artemis.core.server] AMQ221027: Bridge ClusterConnectionBridge@7edd8bf6 [name=$.artemis.internal.sf.my-cluster.3bc7c846-99d0-11ee-839f-02f320d447a2, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.3bc7c846-99d0-11ee-839f-02f320d447a2, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], temp=false]@7082d9ae targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@7edd8bf6 [name=$.artemis.internal.sf.my-cluster.3bc7c846-99d0-11ee-839f-02f320d447a2, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.3bc7c846-99d0-11ee-839f-02f320d447a2, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=amq-broker], temp=false]@7082d9ae targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=amq-broker-ss-1-amq-broker-hdls-svc-demo-svc-cluster-local], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@926188689[nodeUUID=319ec50a-99d0-11ee-b22b-1e89dc031c98, connector=TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=amq-broker-ss-0-amq-broker-hdls-svc-demo-svc-cluster-local, address=, server=ActiveMQServerImpl::name=amq-broker])) [initialConnectors=[TransportConfiguration(name=artemis, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=61616&host=amq-broker-ss-1-amq-broker-hdls-svc-demo-svc-cluster-local], discoveryGroupConfiguration=null]] is connected
   that, if I'm not wrong, confirms the cluster has been created with success.

Can you suggest the right configuration to reach the goal desribed?
From your latest response, I'm not confident I'm applying the right one.

Many thanks in advance.

[brusdev]

Hi @eanbiso, your config LGTM