clusterer_admin.xml

<!-- Module User's Guide -->

<chapter>

	<title>&adminguide;</title>

	<section id="overview" xreflabel="Overview">
	<title>Overview</title>
	<para>
		The <emphasis>clusterer</emphasis> module is used to organize multiple &osips; instances into groups(clusters) in which the nodes can communicate with each other in order to replicate, share information or perform distributed tasks. The distributed logic is performed either by different modules that use the <emphasis>clusterer</emphasis> interface (i.e. the <emphasis>dialog</emphasis> module can replicate dialogs/profiles, the <emphasis>ratelimit</emphasis> module can share pipes across multiple 
		instances etc.) or at the script level. The <emphasis>clusterer</emphasis> module itself only provides an interface to send/receive BIN packets and get notifications about node availability. It achieves this by internally learning the cluster topology and state of the nodes. Provisioning the nodes within a cluster is done over the database or through the configuration script. The node-related information can be checked and triggered to be reloaded by sending commands over the MI interface.
	</para>
	<para>
		The topology established by the <emphasis>clusterer</emphasis> module is an overlay of nodes where the "links" represent communication availability at BIN interface level. For this purpose, a probing mechanism is used, consisting of regular pings to all nodes in a cluster for which replies must be received within a given interval. All nodes in the cluster exchange information about the state of their links with other nodes and compute a "routing table" which gives a next hop for each destination. The metric for the shortest path is the number of hops. When there is no direct link to a destination, the BIN packet sent by a module is transparently routed through the cluster.
	</para>
	<para>
		Note that an &osips; instance can belong to multiple clusters, communicating and establishing the topology separately for each one. In order to provision this in the database or the script, each node has an unique ID at global level, which can be referenced in each cluster.
	</para>
	<para>
		An &osips; instance can dynamically learn all the nodes in the cluster if database provisioning is not desired. It is enough to define at least one neigbour in the script in order to discover all the cluster components.
	</para>
	</section>

	<section id="capabilities" xreflabel="Capabilities layer">
	<title>Capabilities layer</title>
	<para>
		The clusterer module also keeps track of the state of the nodes in terms of data synchronization for the functionalities (or "capabilities") implemented on top by other modules. Some capabilities require a full data sync(at &osips; startup or at runtime via MI) from a valid "donor" node in the cluster that has the full data set. Furthermore, a capability can query the clusterer module in order to partition some distributed logic only over the synchronized nodes in the cluster.
	</para>
	<para>
		Each node in the cluster starts with an empty dataset and tries to find
		a suitable node to pull data from. In order to help "bootstrap" the
		cluster, a "seed" node should be defined. This is done by setting the value
		<emphasis>seed</emphasis> for the <emphasis role="bold">flags</emphasis>
		column in the clusterer table(or the property with the same name in the
		<emphasis>my_node_info</emphasis> parameter). The seed node will simply
		fall back to a "synced" state after a configurable interval(
		<xref linkend="param_seed_fallback_interval"/> parameter). Note that
		this mechanism is required only for capabilities that synchronize data
		at startup, so check the corresponding modules documentation.
	</para>
	<para>
		The clusterer module transparently exposes the <emphasis>sip_addr</emphasis> column from the clusterer table(or the property with the same name in the <emphasis>my_node_info</emphasis> parameter) to the modules on top so check the corresponding modules documentation for the use of this node related information.
	</para>
	</section>

        <section id="dependencies" xreflabel="Dependencies">
	<title>Dependencies</title>
	<section>
		<title>&osips; Modules</title>
		<para>
		The following modules must be loaded before this module:
                    <itemizedlist>
			<listitem>
			<para>
				<emphasis>a database module</emphasis> - if <xref linkend="param_db_mode"/>
				is <emphasis>1</emphasis>.
			</para>
			</listitem>
			<listitem>
			<para>
				<emphasis>proto_bin module</emphasis>.
			</para>
			</listitem>
                    </itemizedlist>
		</para>
	</section>

        <section>
		<title>External Libraries or Applications</title>
		<para>
		The following libraries or applications must be installed before
		running &osips; with this module loaded:
			<itemizedlist>
			<listitem>
			<para>
				<emphasis>None</emphasis>.
			</para>
			</listitem>
			</itemizedlist>
		</para>
	</section>
	</section>

    <section id="exported_parameters" xreflabel="Exported Parameters">
	<title>Exported Parameters</title>

		<section id="param_my_node_id" xreflabel="my_node_id">
            <title><varname>my_node_id</varname></title>
            <para>
				The id of the local instance. This parameter must be equal to one of the
				<emphasis>node_id</emphasis> fields in the database.
			</para>
            <para>
		<emphasis>
			No default value. This parameter must be explicitly set to a value greater than zero.
		</emphasis>
            </para>
            <example>
		<title>Set <varname>my_node_id</varname> parameter</title>
		<programlisting format="linespecific">
...
modparam("clusterer", "my_node_id", 1)
...
		</programlisting>
		</example>
        </section>

        <section id="param_db_mode" xreflabel="db_mode">
            <title><varname>db_mode</varname></title>
            <para>
				Specifies whether the node information for the local instance,
				as well as other instances in the cluster, should be loaded from
				the database or configured in the script(see <xref linkend="param_my_node_info"/>
				and <xref linkend="param_neighbor_node_info"/>). A value of <quote>0</quote>
				means that DB is not used and the cluster topology in terms of node
				information will be discovered dynamically at runtime.
			</para>
			<para>
				If DB mode is enabled, only the nodes defined in the database will be
				accepted by this instance.
			</para>
            <para>
		<emphasis>
			Default value is <quote>1</quote>
		</emphasis>
            </para>
            <example>
		<title>Set <varname>db_mode</varname> parameter</title>
		<programlisting format="linespecific">
...
modparam("clusterer", "db_mode", 0)
...
		</programlisting>
		</example>
        </section>

		<section id="param_db_url" xreflabel="db_url">
            <title><varname>db_url</varname></title>
		<para>
		The database url.
		</para>
		<para>
		<emphasis>
			Default value is <quote>NULL</quote>.
		</emphasis>
		</para>
		<example>
		<title>Set <varname>db_url</varname> parameter</title>
		<programlisting format="linespecific">
...
modparam("clusterer", "db_url",
	"mysql://opensips:opensipsrw@localhost/opensips")
...
		</programlisting>
		</example>
        </section>

        <section id="param_db_table" xreflabel="db_table">
            <title><varname>db_table</varname></title>
            <para>
                The name of the table storing the clustering information.
            </para>
            <para>
		<emphasis>
			Default value is <quote>clusterer</quote>.
		</emphasis>
            </para>
            <example>
		<title>Set <varname>db_table</varname> parameter</title>
		<programlisting format="linespecific">
...
modparam("clusterer", "db_table", "clusterer")
...
		</programlisting>
		</example>
        </section>

		<section id="param_sharing_tag" xreflabel="sharing_tag">
			<title><varname>sharing_tag</varname></title>
			<para>
			The definition of a sharing tag. The sharing tag is 
			managed by the clusterer module, but can be used (in terms
			of reading its state) by any module build on top of 
			clusterer engine, like dialog or presence.
			</para>
			<para>
			Note that ohter tags may be dynamically learned during runtime via 
			clustering communication with other nodes.
			</para>
			<para>
			The format for this value is <quote>tag_name / cluster_id = active/backup</quote>.
			</para>
			<para>
			 Multiple definitions of this parameter are allowed. The default value is <quote>none</quote>.
			</para>
 			<example>
			<title>Set <varname>sharing_tag</varname> parameter</title>
<programlisting format="linespecific">
...
modparam("clusterer", "sharing_tag", "vip1/2=active")
modparam("clusterer", "sharing_tag", "node/10=backup")
...
</programlisting>
			</example>
		</section>

        <section id="param_my_node_info" xreflabel="my_node_info">
            <title><varname>my_node_info</varname></title>
            <para>
				Node specification similar to the information provided by a row in
				the clusterer DB table corresponding to the local instance. This
				parameter can be set multiple times in order to include the local
				node in multiple clusters.
			</para>
			<para>
				Parameter format: multiple "<emphasis>prop=value</emphasis>" property
				definitions separated by '<emphasis>,</emphasis>' where the name of the
				properties is the same as the DB column names. At least the
				<emphasis>cluster_id</emphasis> and <emphasis>url</emphasis>
				properties must be defined.
			</para>
            <para>
			This parameter is required if <xref linkend="param_db_mode"/> is set
			to <quote>0</quote> in order to properly advertise information about
			the local instance in the dynamic node learning process.
            </para>
            <example>
		<title>Set <varname>my_node_info</varname> parameter</title>
		<programlisting format="linespecific">
...
modparam("clusterer", "my_node_info", "cluster_id=1, url=bin:192.168.0.5:5566")
...
		</programlisting>
		</example>
        </section>

        <section id="param_neighbor_node_info" xreflabel="neighbor_node_info">
            <title><varname>neighbor_node_info</varname></title>
            <para>
				Node specification similar to the information provided by a row in
				the clusterer DB table corresponding to another instance in the
				cluster. This node will be the entry point in the cluster for the
				local instance in the dynamic node learning process. This parameter
				can be set multiple times to define multiple neigbors to connect to (or
				the same neighbor but in different clusters).
			</para>
			<para>
				Parameter format: multiple "<emphasis>prop=value</emphasis>" property
				definitions separated by '<emphasis>,</emphasis>' where the name of
				the properties is the same as the DB column names. At least the
				<emphasis>cluster_id</emphasis>, <emphasis>node_id</emphasis>
				and <emphasis>url</emphasis> properties must be defined.
			</para>
            <para>
			This parameter should be set at least once if
			<xref linkend="param_db_mode"/> is set to <emphasis>0</emphasis> in order
			to properly learn the cluster topology. If not set, the only way to learn
			the node topology is by other nodes connecting to the local instance.
            </para>
            <example>
		<title>Set <varname>neighbor_node_info</varname> parameter</title>
		<programlisting format="linespecific">
...
modparam("clusterer", "neighbor_node_info", "cluster_id=1,node_id=2,url=bin:192.168.0.6:5566")
...
		</programlisting>
		</example>
        </section>

        <section id="param_ping_interval" xreflabel="ping_interval">
            <title><varname>ping_interval</varname></title>
            <para>
				The interval in seconds between regular pings sent to a neighbour node.
			</para>
            <para>
		<emphasis>
			Default value is <quote>4</quote>
		</emphasis>
            </para>
            <example>
		<title>Set <varname>ping_interval</varname> parameter</title>
		<programlisting format="linespecific">
...
modparam("clusterer", "ping_interval", 1)
...
		</programlisting>
		</example>
        </section>

        <section id="param_ping_timeout" xreflabel="ping_timeout">
            <title><varname>ping_timeout</varname></title>
            <para>
				The time in milliseconds to wait for a reply to a previously sent ping before retrying or considering the link with the neighbour node down. This is also the interval between successive retries if the send fails.
			</para>
            <para>
		<emphasis>
			Default value is <quote>1000</quote>
		</emphasis>
            </para>
            <example>
		<title>Set <varname>ping_timeout</varname> parameter</title>
		<programlisting format="linespecific">
...
modparam("clusterer", "ping_timeout", 500)
...
		</programlisting>
		</example>
        </section>

        <section id="param_node_timeout" xreflabel="node_timeout">
            <title><varname>node_timeout</varname></title>
            <para>
				The time in seconds to wait before pinging is restarted for a failed node.
			</para>
            <para>
		<emphasis>
			Default value is <quote>60</quote>
		</emphasis>
            </para>
            <example>
		<title>Set <varname>node_timeout</varname> parameter</title>
		<programlisting format="linespecific">
...
modparam("clusterer", "node_timeout", 10)
...
		</programlisting>
		</example>
        </section>

        <section id="param_seed_fallback_interval" xreflabel="seed_fallback_interval">
            <title><varname>seed_fallback_interval</varname></title>
            <para>
                Only relevant for "seed" nodes.  The time, in seconds, to wait
                for a suitable donor node before falling back to a "synced"
                state, following a node restart or an MI cluster sync command.
            </para>
            <para>
		<emphasis>
			Default value is <quote>5</quote>.
		</emphasis>
            </para>
            <example>
		<title>Set <varname>seed_fallback_interval</varname> parameter</title>
		<programlisting format="linespecific">
...
modparam("clusterer", "seed_fallback_interval", 10)
...
		</programlisting>
		</example>
        </section>

        <section id="param_sync_packet_max_size" xreflabel="sync_packet_max_size">
            <title><varname>sync_packet_max_size</varname></title>
            <para>
                The maximum size of the BIN packets sent while doing data synchronization. This is only a suggested value as the actual size of the packets may be slightly larger.
            </para>
            <para>
		<emphasis>
			Default value is <quote>65535</quote>.
		</emphasis>
            </para>
            <example>
		<title>Set <varname>sync_packet_max_size</varname> parameter</title>
		<programlisting format="linespecific">
...
modparam("clusterer", "sync_packet_max_size", 32765)
...
		</programlisting>
		</example>
        </section>

        <section id="param_id_col" xreflabel="id_col">
            <title><varname>id_col</varname></title>
            <para>
                The name of the column storing an id for the table rows.
            </para>
            <para>
		<emphasis>
			Default value is <quote>id</quote>.
		</emphasis>
            </para>
            <example>
		<title>Set <varname>id_col</varname> parameter</title>
		<programlisting format="linespecific">
...
modparam("clusterer", "id_col", "id")
...
		</programlisting>
		</example>
        </section>

        <section id="param_cluster_id_col" xreflabel="cluster_id_col">
            <title><varname>cluster_id_col</varname></title>
            <para>
                The name of the column to store the id of a cluster.
            </para>
            <para>
		<emphasis>
			Default value is <quote>cluster_id</quote>.
		</emphasis>
            </para>
            <example>
		<title>Set <varname>cluster_id_col</varname> parameter</title>
		<programlisting format="linespecific">
...
modparam("clusterer", "cluster_id_col", "cluster_id")
...
		</programlisting>
		</example>
        </section>

        <section id="param_node_id_col" xreflabel="node_id_col">
            <title><varname>node_id_col</varname></title>
            <para>
                The name of the column to store the id of an instance. The values must be greater than 0.
            </para>
            <para>
		<emphasis>
			Default value is <quote>node_id</quote>.
		</emphasis>
            </para>
            <example>
		<title>Set <varname>node_id_col</varname> parameter</title>
		<programlisting format="linespecific">
...
modparam("clusterer", "node_id_col", "node_id")
...
		</programlisting>
		</example>
        </section>

        <section id="param_url_col" xreflabel="url_col">
            <title><varname>url_col</varname></title>
            <para>
                The name of the column containing the instance url. The values must be greater than 0.
            </para>
            <para>
		<emphasis>
			Default value is <quote>url</quote>.
		</emphasis>
            </para>
            <example>
		<title>Set <varname>url_col</varname> parameter</title>
		<programlisting format="linespecific">
...
modparam("clusterer", "url_col", "url")
...
		</programlisting>
		</example>
        </section>

        <section id="param_state_col" xreflabel="state_col">
            <title><varname>state_col</varname></title>
            <para>
                The name of the column storing the state of the node(enabled/disabled).
            </para>
            <para>
		<emphasis>
			Default value is <quote>state</quote>.
		</emphasis>
            </para>
            <example>
		<title>Set <varname>state_col</varname> parameter</title>
		<programlisting format="linespecific">
...
modparam("clusterer", "state_col", "state")
...
		</programlisting>
		</example>
        </section>

        <section id="param_no_ping_retries_col" xreflabel="no_ping_retries_col">
            <title><varname>no_ping_retries_col</varname></title>
            <para>
                The name of the column containing the maximum number of ping retries before the link with the neighbour node is considered down.
            </para>
            <para>
		<emphasis>
			Default value is <quote>no_ping_retries</quote>.
		</emphasis>
            </para>
            <example>
		<title>Set <varname>no_ping_retries_col</varname> parameter</title>
		<programlisting format="linespecific">
...
modparam("clusterer", "no_ping_retries_col", "no_ping_retries")
...
		</programlisting>
		</example>
        </section>

        <section id="param_priority_col" xreflabel="priority_col">
            <title><varname>priority_col</varname></title>
            <para>
                The name of the column storing the node priority to be chosen as next hop in case of same length(number of hops) paths when rerouting messages.
            </para>
            <para>
		<emphasis>
			Default value is <quote>priority</quote>.
		</emphasis>
            </para>
            <example>
		<title>Set <varname>priority_col</varname> parameter</title>
		<programlisting format="linespecific">
...
modparam("clusterer", "priority_col", "priority")
...
		</programlisting>
		</example>
        </section>

        <section id="param_sip_addr_col" xreflabel="sip_addr_col">
            <title><varname>sip_addr_col</varname></title>
            <para>
                The name of the column containing a SIP address for the node.
            </para>
            <para>
		<emphasis>
			Default value is <quote>sip_addr</quote>.
		</emphasis>
            </para>
            <example>
		<title>Set <varname>sip_addr_col</varname> parameter</title>
		<programlisting format="linespecific">
...
modparam("clusterer", "sip_addr_col", "sip_addr")
...
		</programlisting>
		</example>
        </section>

        <section id="param_flags_col" xreflabel="flags_col">
            <title><varname>flags_col</varname></title>
            <para>
                The name of the column containing the node flags.
            </para>
            <para>
		<emphasis>
			Default value is <quote>flags</quote>.
		</emphasis>
            </para>
            <example>
		<title>Set <varname>flags_col</varname> parameter</title>
		<programlisting format="linespecific">
...
modparam("clusterer", "flags_col", "flags")
...
		</programlisting>
		</example>
        </section>

        <section id="param_description_col" xreflabel="description_col">
            <title><varname>description_col</varname></title>
            <para>
                The name of the column containing a node description.
            </para>
            <para>
		<emphasis>
			Default value is <quote>description</quote>.
		</emphasis>
            </para>
            <example>
		<title>Set <varname>description_col</varname> parameter</title>
		<programlisting format="linespecific">
...
modparam("clusterer", "description_col", "description")
...
		</programlisting>
		</example>
        </section>

        </section>

        <section id="exported_functions" xreflabel="exported_functions">
		<title>Exported Functions</title>
			<section id="func_cluster_send_req" xreflabel="cluster_send_req()">
				<title>
				<function moreinfo="none">cluster_send_req(cluster_id, dst_id, msg, [tag])</function>
				</title>
				<para>
					This function is used to send a generic, request-like message, containing custom data, to a specific node in a cluster, directly from the script. The message is not a "request" per se but according to the logic on the receiving side, that node can send back a reply. In order to correlate a received reply with the request sent out, the function returns, through the <emphasis>tag</emphasis> parameter, a randomly generated communication tag, which is sent along in the the original message, that can be checked against the tag received in a reply.
				</para>
				<para>
					Meaning of the parameters is as follows:
				<itemizedlist>
					<listitem>
						<para><emphasis>cluster_id</emphasis> (int) - the cluster ID of the destination node;</para>
					</listitem>
					<listitem>
						<para><emphasis>dst_id</emphasis> (int) - the ID of the destiantion node;</para>
					</listitem>
					<listitem>
						<para><emphasis>msg</emphasis> (string) - actual message payload;</para>
					</listitem>
					<listitem>
						<para><emphasis>tag</emphasis> (var, optional) - randomly generated communication tag.</para>
					</listitem>
				</itemizedlist>
				</para>
				<para>
					The function can return the following values:
					<itemizedlist>
						<listitem>
							<para><emphasis>1</emphasis> - successfully sent message to destination node or a valid next hop</para>
						</listitem>
						<listitem>
							<para><emphasis>-1</emphasis> - local node is disabled so sending is impossbile</para>
						</listitem>
						<listitem>
							<para><emphasis>-2</emphasis> - destination node is not reachable through any path according to the discovered topology</para>
						</listitem>
						<listitem>
							<para><emphasis>-3</emphasis> - destination node or valid next hop appear to be reachable but send failed or other &osips; internal error</para>
						</listitem>
					</itemizedlist>
				</para>
				<para>
					This function can be used from REQUEST_ROUTE, FAILURE_ROUTE, ONREPLY_ROUTE, BRANCH_ROUTE, LOCAL_ROUTE and EVENT_ROUTE.
				</para>

				<example>
						<title>cluster_send_req() usage</title>
						<programlisting format="linespecific">
...
# send a request
cluster_send_req(1, 1, "Check USER: $fU", $var(req_tag));
# wait for reply
$avp(filter) = "tag=" + $var(req_tag);
async(wait_for_event("E_CLUSTERER_RPL_RECEIVED", $avp(filter), 5), rpl_resume);
# done
...
route[rpl_resume] {
  xlog("Received reply: $avp(msg)\n");
}
...
				</programlisting>
				</example>
			</section>
			<section id="func_cluster_send_rpl" xreflabel="cluster_send_rpl()">
				<title>
				<function moreinfo="none">cluster_send_rpl(cluster_id, dst_id, msg, tag)</function>
				</title>
				<para>
					This function is used to send a generic, reply-like message, containing custom data, to a specific node in a cluster, directly from the script. The message is marked as a "reply" so this function should ony be used for replying to a previously request-like message received. In order for the other node, which initially sent a request, to be able to correlate it with this reply, a communication tag, received along with the request, should be passed to the function.
				</para>
				<para>
					Meaning of the parameters is as follows:
				<itemizedlist>
					<listitem>
						<para><emphasis>cluster_id</emphasis> (int) - the cluster ID of the destination node;</para>
					</listitem>
					<listitem>
						<para><emphasis>dst_id</emphasis> (int) - the ID of the destiantion node;</para>
					</listitem>
					<listitem>
						<para><emphasis>msg</emphasis> (string) - actual message payload;</para>
					</listitem>
					<listitem>
						<para><emphasis>tag</emphasis> (var) - communication tag.</para>
					</listitem>
				</itemizedlist>
				</para>
				<para>
					The function can return the following values:
					<itemizedlist>
						<listitem>
							<para><emphasis>1</emphasis> - successfully sent message to destination node or a valid next hop</para>
						</listitem>
						<listitem>
							<para><emphasis>-1</emphasis> - local node is disabled so sending is impossbile</para>
						</listitem>
						<listitem>
							<para><emphasis>-2</emphasis> - destination node is not reachable through any path according to the discovered topology</para>
						</listitem>
						<listitem>
							<para><emphasis>-3</emphasis> - destination node or valid next hop appear to be reachable but send failed or other &osips; internal error</para>
						</listitem>
					</itemizedlist>
				</para>
				<para>
					This function can be used from REQUEST_ROUTE, FAILURE_ROUTE, ONREPLY_ROUTE, BRANCH_ROUTE, LOCAL_ROUTE and EVENT_ROUTE.
				</para>
				<example>
						<title>cluster_send_rpl() usage</title>
						<programlisting format="linespecific">
...
event_route[E_CLUSTERER_REQ_RECEIVED] {
  cluster_send_rpl($param(cluster_id), $param(src_id), $var(my_reply), $param(tag));
}
...
				</programlisting>
				</example>
			</section>

			<section id="func_cluster_broadcast_req" xreflabel="cluster_broadcast_req()">
				<title>
				<function moreinfo="none">cluster_broadcast_req(cluster_id, msg, [tag])</function>
				</title>
				<para>
					This function has a similar behaviour to the <function moreinfo="none">cluster_send_req()</function> function with the exception that the message is sent to all the nodes in the specified cluster.
				</para>
				<para>
					The function can return the following values:
					<itemizedlist>
						<listitem>
							<para><emphasis>1</emphasis> - successfully sent message to at least one node;</para>
						</listitem>
						<listitem>
							<para><emphasis>-1</emphasis> - local node is disabled so sending is impossbile;</para>
						</listitem>
						<listitem>
							<para><emphasis>-2</emphasis> - all nodes in the cluster are unreachable according to the discovered topology;</para>
						</listitem>
						<listitem>
							<para><emphasis>-3</emphasis> - send failed for all nodes in the cluster or other &osips; internal error.</para>
						</listitem>
					</itemizedlist>
				</para>
				<para>
					The meaning of the parameters is the same as for <function moreinfo="none">cluster_send_req()</function>.
				</para>
				<para>
					This function can be used from REQUEST_ROUTE, FAILURE_ROUTE, ONREPLY_ROUTE, BRANCH_ROUTE, LOCAL_ROUTE and EVENT_ROUTE.
				</para>
				<example>
						<title>cluster_broadcast_req() usage</title>
						<programlisting format="linespecific">
...
cluster_broadcast_req($var(cl_id), $var(share_data));
...
				</programlisting>
				</example>
			</section>

			<section id="func_cluster_check_addr" xreflabel="cluster_check_addr()">
				<title>
				<function moreinfo="none">cluster_check_addr(cluster_id, ip, addr_type)</function>
				</title>
				<para>
					This function checks whether the given IP address belongs
					to one of the nodes in the cluster.
				</para>
				<para>Parameters:</para>
				<itemizedlist>
					<listitem><para>
						<emphasis>cluster_id</emphasis> (int)
					</para></listitem>
					<listitem><para>
						<emphasis>ip</emphasis> (string)
					</para></listitem>
					<listitem><para>
						<emphasis>addr_type</emphasis> (string, optional) -
						select the address of the node that the comparison
						is made against, with the possible values of:
						<itemizedlist>
							<listitem>
								<para><emphasis>"sip"</emphasis> (default) - a node's DB provisioned SIP address</para>
							</listitem>
							<listitem>
								<para><emphasis>"bin"</emphasis> - a node's BIN interface listener</para>
							</listitem>
						</itemizedlist>
					</para></listitem>
				</itemizedlist>
				<para>
					This function can be used from REQUEST_ROUTE, FAILURE_ROUTE, ONREPLY_ROUTE, BRANCH_ROUTE, LOCAL_ROUTE and EVENT_ROUTE.
				</para>
				<example>
						<title>cluster_check_addr() usage</title>
						<programlisting format="linespecific">
...
if (cluster_check_addr(1, $si)) {
	...
}
...
				</programlisting>
				</example>
			</section>
        </section>

        <section id="exported_mi_functions" xreflabel="Exported MI Functions">
	<title>Exported MI Functions</title>
	<section id="mi_clusterer_reload" xreflabel="clusterer_reload">
		<title>
		<function moreinfo="none">clusterer_reload</function>
		</title>
		<para>
        Reloads data from the clusterer database. The currently established topology will be lost and the node will rediscover the new topology.
		</para>
		<para>
		Name: <emphasis>clusterer_reload</emphasis>
		</para>
		<para>Parameters:<emphasis>none</emphasis> </para>
 		<para>
		MI FIFO Command Format:
		</para>
		<programlisting  format="linespecific">
		opensips-cli -x mi clusterer_reload
		</programlisting>
	</section>

        <section id="mi_clusterer_list" xreflabel="clusterer_list">
		<title>
		<function moreinfo="none">clusterer_list</function>
		</title>
		<para>
			Lists information(node id, URL, link state with that node etc.) about the other nodes in each cluster.
		</para>
		<para>
		Name: <emphasis>clusterer_list</emphasis>
		</para>
		<para>Parameters:<emphasis>none</emphasis> </para>
		<example>
		<title><function>clusterer_list</function> usage</title>
		<programlisting format="linespecific">
$ opensips-cli -x mi clusterer_list
{
    "Clusters": [
        {
            "cluster_id": 1,
            "Nodes": [
                {
                    "node_id": 1,
                    "db_id": 1,
                    "url": "bin:127.0.0.1",
                    "link_state": "Up",
                    "next_hop": "1",
                    "description": "none"
                }
            ]
        }
    ]
}
</programlisting>
		</example>
	</section>

	<section id="mi_clusterer_list_topology" xreflabel="clusterer_list_topology">
		<title>
		<function moreinfo="none">clusterer_list_topology</function>
		</title>
		<para>
			Lists each cluster's topology from the local node's perspective as an adjacency list. A node appears as a neighbour if the link with that node is up.
		</para>
		<para>
			Note that if a node id appears in multiple clusters, it refers to the same instance that belongs to different clusters, for which it has a different topology.
		</para>
		<para>
		Name: <emphasis>clusterer_list_topology</emphasis>
		</para>
		<para>Parameters:<emphasis>none</emphasis> </para>
		<example>
		<title><function>clusterer_list_topology</function> usage</title>
		<programlisting format="linespecific">
$ opensips-cli -x mi clusterer_list_topology
{
    "Clusters": [
        {
            "cluster_id": 1,
            "Nodes": [
                {
                    "node_id": 2,
                    "Neighbours": [
                        1
                    ]
                },
                {
                    "node_id": 1,
                    "Neighbours": [
                        2
                    ]
                }
            ]
        }
    ]
}
</programlisting>
		</example>
	</section>

        <section id="mi_clusterer_set_status" xreflabel="clusterer_set_status">
		<title>
		<function moreinfo="none">clusterer_set_status</function>
		</title>
		<para>
            Sets the status(Enabled/Disabled) of a node. If the local instance is disabled, the node will not send any messages and ignore received ones thus appearing as a failed node in the topology (from the other node's perspective). If a different node is disabled, the specified node will simply be ignored by the local instance in terms of sending/receiving any messages, as if no longer part of the topology.
        </para>
		<para>
		Name: <emphasis>clusterer_set_status</emphasis>
		</para>
		<para>Parameters:</para>
		<itemizedlist>
			<listitem><para>
			<emphasis>cluster_id</emphasis> - indicates the id of the cluster.
			</para></listitem>
			<listitem><para>
			<emphasis>node_id</emphasis> (optional) - indicates the id of the node to be disabled.
			If missing, the local instance will be disalbed.
			</para></listitem>
            <listitem><para>
			<emphasis>status</emphasis> - indicates the new status(0 - Disabled, 1 - Enabled).
			</para></listitem>
		</itemizedlist>
 		<para>
		MI FIFO Command Format:
		</para>
		<programlisting  format="linespecific">
		#disable the local instance
		opensips-cli -x mi clusterer_set_status 1 0
		#disable node ID 3
		opensips-cli -x mi clusterer_set_status 1 3 0
		</programlisting>
	</section>

		<section id="mi_clusterer_remove_node" xreflabel="clusterer_remove_node">
		<title>
		<function moreinfo="none">clusterer_remove_node</function>
		</title>
		<para>
			Removes a node from the cluster's topology. It is enough to run the function
			on a single node in order to remove the target node from all the other
			nodes in the cluster. If the node to be removed is running when triggering
			this function, it will be automatically disabled (equivalent to running
			<xref linkend="mi_clusterer_set_status"/> on that specific node).
        </para>
        <para>
			This function can only be used when <xref linkend="param_db_mode"/> is set to
			<emphasis>0</emphasis> (disabled).
        </para>
		<para>
		Name: <emphasis>clusterer_remove_node</emphasis>
		</para>
		<para>Parameters:</para>
		<itemizedlist>
			<listitem><para>
			<emphasis>cluster_id</emphasis> - cluster ID
			</para></listitem>
            <listitem><para>
			<emphasis>node_id</emphasis> - ID of the node to be removed.
			</para></listitem>
		</itemizedlist>
		<para>
		MI FIFO Command Format:
		</para>
		<programlisting  format="linespecific">
		opensips-cli -x mi clusterer_remove_node 1 3
		</programlisting>
	</section>

		<section id="mi_cluster_send_mi" xreflabel="cluster_send_mi">
		<title>
		<function moreinfo="none">cluster_send_mi</function>
		</title>
		<para>
        Dispatches a given MI command to be run on a specific node in the cluster.
        </para>
		<para>
		Name: <emphasis>cluster_send_mi</emphasis>
		</para>
		<para>Parameters:</para>
		<itemizedlist>
			<listitem><para>
			<emphasis>cluster_id</emphasis> - id of the cluster.
			</para></listitem>
            <listitem><para>
			<emphasis>destination</emphasis> - id of the destination node
			</para></listitem>
			<listitem><para>
			<emphasis>cmd_name</emphasis> - name of the MI command to be run
			</para></listitem>
			<listitem><para>
			<emphasis>cmd_params</emphasis> (optional) - array of parameters for
			the MI command to be run
			</para></listitem>
		</itemizedlist>
		<para>
			Note that MI commands that require named parameters or arrays as
			parameter values are not currently supported.
		</para>
		<para>
		MI FIFO Command Format:
		</para>
		<programlisting  format="linespecific">
opensips-cli -x mi cluster_send_mi 1 3 lb_reload
		</programlisting>
		</section>

		<section id="mi_cluster_broadcast_mi" xreflabel="cluster_broadcast_mi">
		<title>
		<function moreinfo="none">cluster_broadcast_mi</function>
		</title>
		<para>
        Dispatches a given MI command to be run on all the nodes in a cluster. The command is also executed locally.
        </para>
		<para>
		Name: <emphasis>cluster_broadcast_mi</emphasis>
		</para>
		<para>Parameters:</para>
		<itemizedlist>
			<listitem><para>
			<emphasis>cluster_id</emphasis> - id of the cluster.
			</para></listitem>
			<listitem><para>
			<emphasis>cmd_name</emphasis> - name of the MI command to be run
			</para></listitem>
			<listitem><para>
			<emphasis>cmd_params</emphasis> (optional) - array of parameters for
			the MI command to be run
			</para></listitem>
		</itemizedlist>
		<para>
			Note that MI commands that require named parameters or arrays as
			parameter values are not currently supported.
		</para>
		<para>
		MI FIFO Command Format:
		</para>
		<programlisting  format="linespecific">
opensips-cli -x mi cluster_broadcast_mi 1 dr_reload partition_5
		</programlisting>
		</section>

		<section id="mi_clusterer_list_cap" xreflabel="clusterer_list_cap">
		<title>
		<function moreinfo="none">clusterer_list_cap</function>
		</title>
		<para>
			Lists the registered capabilities and their states.
		</para>
		<para>
		Name: <emphasis>clusterer_list_cap</emphasis>
		</para>
		<para>Parameters:<emphasis>none</emphasis> </para>
		<example>
		<title><function>clusterer_list_cap</function> usage</title>
		<programlisting format="linespecific">
$ opensips-cli -x mi clusterer_list_cap
{
    "Clusters": [
        {
            "cluster_id": 1,
            "Capabilities": [
                {
                    "name": "dialog-dlg-repl",
                    "state": "Ok",
                    "enabled": "yes"
                },
                {
                    "name": "dialog-prof-repl",
                    "state": "Ok",
                    "enabled": "yes"
                }
            ]
        }
    ]
}
</programlisting>
		</example>
		</section>

		<section id="mi_clusterer_set_cap_status" xreflabel="clusterer_set_cap_status">
		<title>
		<function moreinfo="none">clusterer_set_cap_status</function>
		</title>
		<para>
            Sets the status(Enabled/Disabled) of a capability. If a capability is disabled, the node will not send any replication/sync messages belonging to that capability. Likewise, received messages will be dropped. Also, the cabability will transition to a "not synced" state and the node will no longer be able to be a donor for syncing.
        </para>
		<para>
		Name: <emphasis>clusterer_set_cap_status</emphasis>
		</para>
		<para>Parameters:</para>
		<itemizedlist>
			<listitem><para>
			<emphasis>cluster_id</emphasis> - the id of the cluster
			</para></listitem>
			<listitem><para>
			<emphasis>capability</emphasis> - name of the capability, as listed by
			<xref linkend="mi_clusterer_list_cap"/>
			</para></listitem>
            <listitem><para>
			<emphasis>status</emphasis> - indicates the new status(0 - Disabled, 1 - Enabled).
			</para></listitem>
		</itemizedlist>
		<para>
		MI FIFO Command Format:
		</para>
		<programlisting  format="linespecific">
		#disable dialog replication in cluster 1
		opensips-cli -x mi clusterer_set_cap_status 1 dialog-dlg-repl 0
		#enable dialog profile replication in cluster 2
		opensips-cli -x mi clusterer_set_cap_status 2 dialog-prof-repl 1
		</programlisting>
	</section>

		<section id='mi_clusterer_shtag_set_active' xreflabel="clusterer_shtag_set_active">
		<title><function moreinfo="none">clusterer_shtag_set_active</function></title>
		<para>
		Set the given sharing tag to the <emphasis>active</emphasis> state.
		The information about this change is also broadcasted in the cluster 
		in order to force any other node that may be active on this tag to 
		step down to backup.
		</para>
		<para>
		Name: <emphasis>clusterer_shtag_set_active</emphasis>
		</para>
		<para>Parameters: <emphasis>tag</emphasis> - the name of
		the tag to be set active and the cluster it belogs to, in the
		format 'tag/cluster_id'.
		</para>
		<para>
		MI FIFO Command Format:
		</para>
		<programlisting  format="linespecific">
		opensips-cli -x mi clusterer_shtag_set_active vip1/3
		</programlisting>
		</section>

		<section id="mi_clusterer_list_shtags" xreflabel="clusterer_list_shtags">
		<title><function moreinfo="none">clusterer_list_shtags</function></title>
		<para>
		Lists all known sharing tags and their states.
		</para>
		<para>
		Name: <emphasis>clusterer_list_shtags</emphasis>
		</para>
		<para>Parameters: <emphasis>Command takes no parameters</emphasis>
		</para>
		<para>
		MI FIFO Command Format:
		</para>
		<programlisting  format="linespecific">
		opensips-cli -x mi clusterer_list_shtags
		</programlisting>
	</section>

</section>


	<section id="exported_variables">
	<title>Exported Script Variables</title>
		<section id="var_cluster_sh_tag" xreflabel="$cluster.sh_tag">
			<title><varname>$cluster.sh_tag</varname></title>
			<para>
			This is a read/write variable that allows access to the
			sharing tags managed by the clusterer module.
			</para>
			<para>
			The name of such a variable has the format of 
			<emphasis>tag_name/cluster_id</emphasis>, like 
			<emphasis>$cluster.sh_tag(vip/3)</emphasis> accessing the
			sharing tag "vip" from cluster ID 3.
			</para>
			<para>
			When setting, a sharing tag may be only switched to active by
			assigned it:
			<itemizedlist>
				<listitem>
					<emphasis>"active"</emphasis> string value
				</listitem>
				<listitem>
					<emphasis>1</emphasis> or higher numerical value
				</listitem>
			</itemizedlist>
			</para>
			<para>
			When reading it value, a sharing tag returns:
			<itemizedlist>
				<listitem>
					<emphasis>"active" or 1</emphasis> if active
				</listitem>
				<listitem>
					<emphasis>"backup" or 0</emphasis> if backup
				</listitem>
			</itemizedlist>
			A NULL value may returned only as a result of an internal error
			(like memory errors).
			</para>
		</section>
	</section>

<section id="exported_events" xreflabel="Exported Events">
<title>Exported Events</title>
	<section id="event_E_CLUSTERER_REQ_RECEIVED" xreflabel="E_CLUSTERER_REQ_RECEIVED">
		<title>
		<function moreinfo="none">E_CLUSTERER_REQ_RECEIVED</function>
		</title>
		<para>
			This event is raised when a generic, request-like, clusterer message is received. This type of message is sent directly from the script and not by an &osips; module.
		</para>
		<para>Parameters:</para>
		<itemizedlist>
			<listitem><para>
					<emphasis>cluster_id</emphasis> - The cluster ID of the source node.
			</para></listitem>
			<listitem><para>
					<emphasis>src_id</emphasis> - The ID of the source node.
			</para></listitem>
			<listitem><para>
				<emphasis>msg</emphasis> - The actual message payload.
			</para></listitem>
			<listitem><para>
				<emphasis>tag</emphasis> - The communication tag of this message, generated by the source node. This could be used to send a reply corresponding to the received message by providing the tag to the <function moreinfo="none">cluster_send_rpl()</function> function.
			</para></listitem>
		</itemizedlist>
	</section>

	<section id="event_E_CLUSTERER_RPL_RECEIVED" xreflabel="E_CLUSTERER_RPL_RECEIVED">
		<title>
		<function moreinfo="none">E_CLUSTERER_RPL_RECEIVED</function>
		</title>
		<para>
			This event is raised when a generic, reply-like, clusterer message is received. This type of message is sent directly from the script and not by an &osips; module.
		</para>
		<para>Parameters:</para>
		<itemizedlist>
			<listitem><para>
					<emphasis>cluster_id</emphasis> - The cluster ID of the source node.
			</para></listitem>
			<listitem><para>
					<emphasis>src_id</emphasis> - The ID of the source node.
			</para></listitem>
			<listitem><para>
				<emphasis>msg</emphasis> - The actual message payload.
			</para></listitem>
			<listitem><para>
				<emphasis>tag</emphasis> - The communication tag of this message. This could be used to match the received reply with a request sent with the <function moreinfo="none">cluster_send_req()</function> or <function moreinfo="none">cluster_broadcast_req()</function> functions.
			</para></listitem>
		</itemizedlist>
	</section>

	<section id="event_E_CLUSTERER_NODE_STATE_CHANGED" xreflabel="E_CLUSTERER_NODE_STATE_CHANGED">
		<title>
		<function moreinfo="none">E_CLUSTERER_NODE_STATE_CHANGED</function>
		</title>
		<para>
			This event is raised when the state of a node changes in terms of
			availability.
		</para>
		<para>Parameters:</para>
		<itemizedlist>
			<listitem><para>
					<emphasis>cluster_id</emphasis> - The cluster ID.
			</para></listitem>
			<listitem><para>
					<emphasis>node_id</emphasis> - The ID of the node.
			</para></listitem>
			<listitem><para>
				<emphasis>new_state</emphasis> - The new state of the node, with
				the possible values: 0 - down, 1 - up.
			</para></listitem>
		</itemizedlist>
	</section>

	<section id="event_E_CLUSTERER_SHARING_TAG_CHANGED" xreflabel="E_CLUSTERER_SHARING_TAG_CHANGED">
		<title>
		<function moreinfo="none">E_CLUSTERER_SHARING_TAG_CHANGED</function>
		</title>
		<para>
			This event is raised when the state of a sharing tag changes.
		</para>
		<para>Parameters:</para>
		<itemizedlist>
			<listitem><para>
				<emphasis>name</emphasis> - The name of the sharing tag.
			</para></listitem>
			<listitem><para>
				<emphasis>cluster_id</emphasis> - The cluster ID.
			</para></listitem>
			<listitem><para>
				<emphasis>node_id</emphasis> - The ID of the node.
			</para></listitem>
			<listitem><para>
				<emphasis>state</emphasis> - The new state of the sharing tag,
				the possible values: "active" or "backup".
			</para></listitem>
			<listitem><para>
				<emphasis>reason</emphasis> - short text describing what
				triggered the change of the state, like a another node
				stepping as active, an MI command or script variable.
			</para></listitem>
		</itemizedlist>
	</section>

</section>

<section id="sr_identifiers" xreflabel="Status/Report Identifiers">
	<title>Provided Status/Report Identifiers</title>

	<para>
	The module provides the "clusterer" Status/Report group used so
	far only for reporting purposes.
	</para>
	<para>
	The "sharing_tags" identifier is provided for reporting state changes
	of the sharing_tags (between active and backup), along with the reason
	of the change. This identifier have a 200 records history before 
	discarding the old ones.
	</para>
	<para>
	For how to access and use the Status/Report information, please see
	<ulink url='>https://www.opensips.org/Documentation/Interface-StatusReport-3-3'>https://www.opensips.org/Documentation/Interface-StatusReport-3-3</ulink>.
	</para>

</section>



<section>
	<title>Usage Example</title>
	<para> This section provides an usage example for replicating ratelimit
		pipes between two &osips; instances. It uses the clusterer module to
		manage the replicating nodes, and along with the
		<emphasis>proto_bin</emphasis> module, to send the replicated information.
	</para>
	<para> The setup topology is simple: we have two &osips; nodes running on
		two separate machines (although they could run on the same machine as
		well): <emphasis>Node A</emphasis> has IP 192.168.0.5 and
		<emphasis>Node B</emphasis> has IP 192.168.0.6. Both have, besides the
		traffic listeners (UDP, TCP, etc.), BIN listeners bound on port
		<emphasis>5566</emphasis>. These listeners will be used for the binary
		communication.
	</para>
	<para>
		We insert in the the <emphasis>clusterer</emphasis> table the following:
	</para>
	<example>
		<title>Example database content - clusterer table</title>
		<programlisting format="linespecific">
+----+------------+---------+----------------------+-------+-----------------+----------+----------+-------+-------------+
| id | cluster_id | node_id | url                  | state | no_ping_retries | priority | sip_addr | flags | description |
+----+------------+---------+----------------------+-------+-----------------+----------+----------+-------+-------------+
| 10 |          1 |       1 | bin:192.168.0.5:5566 |     1 |                3|       50 | NULL     | NULL  | Node A      |
| 20 |          1 |       2 | bin:192.168.0.6:5566 |     1 |                3|       50 | NULL     | NULL  | Node B      |
+----+------------+---------+----------------------+-------+-----------------+----------+----------+-------+-------------+
		</programlisting>
	</example>
	<para>
		<itemizedlist>
			<listitem>
				<para><quote>cluster_id</quote> - identifier of the cluster. All nodes within a
					group/cluster should have the same id (in our example,
					both nodes have ID <emphasis>1</emphasis>). The values must be greater than 0.
				</para>
			</listitem>
			<listitem>
				<para><quote>node_id</quote> - identifier of the machine/node so each instance within a
					cluster should have a different ID. The values must be greater than 0. In our example,
					<emphasis>Node A</emphasis> will have ID 1, and
					<emphasis>Node B</emphasis> ID 2.
				</para>
			</listitem>
			<listitem>
				<para><quote>url</quote> - address where all the BIN packets for that instance will be
				sent to.
				</para>
			</listitem>
			<listitem>
				<para><quote>state</quote> - state of the node: <emphasis>1</emphasis> means Enabled,
				<emphasis>0</emphasis> means Disabled. A disabled node will not send any BIN packets
				and will drop received ones.
				</para>
			</listitem>
			<listitem>
				<para><quote>no_ping_retries</quote> - maximum number of ping retries before the link
				with a node is considered down.
				</para>
			</listitem>
			<listitem>
				<para><quote>priority</quote> - the priority of a node to be chosen
				as next hop in case of same length(number of hops) paths when rerouting messages;
				it is not relevant for this two-node topology example.
				</para>
			</listitem>
			<listitem>
				<para><quote>sip_addr</quote> - SIP address for the node that is transparently
				provided to modules; it has no use for the ratelimit module in our example.
				</para>
			</listitem>
			<listitem>
				<para><quote>flags</quote> - used to define a seed node; it has no use in our example.
				</para>
			</listitem>
			<listitem>
				<para><quote>description</quote> - an opaque value used to
					describe the node
				</para>
			</listitem>
		</itemizedlist>
	</para>
	<para>
		After provisioning the two nodes in the database, we have to configure
		the two instances of &osips;. First, we configure <emphasis>Node
			A</emphasis>:
	</para>
	<example>
		<title><emphasis>Node A</emphasis> configuration</title>
		<programlisting format="linespecific">
...
socket= bin:192.168.0.5:5566 # bin listener for Node A

loadmodule "proto_bin.so"

loadmodule "clusterer.so"
modparam("clusterer", "db_url", "mysql://opensips@192.168.0.7/opensips")
modparam("clusterer", "my_node_id", 1) # node_id for Node A

loadmodule "ratelimit.so"
modparam("ratelimit", "pipe_replication_cluster", 1)
...
		</programlisting>
	</example>
	<para>
		Similarly, the configuration for <emphasis>Node B</emphasis> is as follows:
	</para>
	<example>
		<title><emphasis>Node B</emphasis> configuration</title>
		<programlisting format="linespecific">
...
socket= bin:192.168.0.6:5566 # bin listener for Node B

loadmodule "proto_bin.so"

loadmodule "clusterer.so"
# ideally, use the same database for both nodes
modparam("clusterer", "db_url", "mysql://opensips@192.168.0.7/opensips")
modparam("clusterer", "my_node_id", 2) # node_id for Node B

loadmodule "ratelimit.so"
modparam("ratelimit", "pipe_replication_cluster", 1)
...
		</programlisting>
	</example>
</section>

<para>
	Starting the two &osips; instances with the above configurations provides
	your platform the ability to used shared ratelimit pipes in a very
	efficient and scalable way.
</para>

</chapter>