Doris-Connector-JMX.md

Monitor Doris Kafka Connector using Java Management Extensions (JMX)

This article describes how to monitor the Doris Kafka Connector using Java Management Extensions (JMX).

Kafka Connect provides some default JMX Metrics, and Doris Kafka Connector also extends these Metrics to obtain more monitoring indicators. You can also import these indicators into third-party monitoring tools, including (Prometheus, Grafana).

Configure JMX

The JMX Metrics function is enabled by default. If you need to disable this function, you can set jmx=false in the parameters.

1. Enable JMX to connect to your Kafka installation

   • Establish a connection with the remote server. Please append the following content to the KAFKA_JMX_OPTS parameter in the Kafka Connect startup script:

     export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true
      -Dcom.sun.management.jmxremote.authenticate=false
      -Dcom.sun.management.jmxremote.ssl=false
      -Djava.rmi.server.hostname=<ip_address>
      -Dcom.sun.management.jmxremote.port=<jmx_port>"
     

     <ip_address> is the current Kafka Connect host address

     <jmx_port> specifies the port to listen to JMX (please ensure that the port is not occupied)

   • Establish a connection on the same server. Please add the JMX_PORT parameter in the Kafka Connect startup script

     export JMX_PORT=<port_number>
     

2. Restart Kafka Connect

Use Doris Kafka Connector to manage Beans (MBeans)

Kafka Connector Doris provides MBeans for accessing management objects. Through these MBeans indicators, you can clearly understand the internal status of the connector.

The general format of Kafka Connector MBean object names is:

kafka.connector.doris:connector={connector_name}, task={task_id},category={category_name},name={metric_name}

connector={connector_name} The name specified in the Kafka Connect configuration file
{task_id} The task_id assigned by default when Kafka Connect starts
{category_name} specifies the name of the MBean. Each MBean contains specific indicators.
{metric_name} metric name

The following is the Category and Metrics description of Doris Kafka Connector:

Category	Metric Name	Data Type	Description
offsets	committed-offset	long	Successfully submitted to Doris' offset
total-processed	total-load-count	long	The total number of data successfully imported to doris through stream-load (or the total number of data files uploaded through copy-into)
	total-record-count	long	Total number of records processed cumulatively
	total-data-size	long	Cumulative processed data size (unit: byte)
buffer	buffer-record-count	long	The number of Records in the current buffer when flushing Doris
	buffer-size-bytes	long	The size of bytes in the current buffer when flushing Doris
	buffer-memory-usage	long	The memory size currently occupied by buffer (unit: byte)

For information about Kafka Connect’s default JMX Metrics, please refer to: Monitoring Kafka Connect and Connectors