From 607807d01f591e61eafeb595da2247866a09336c Mon Sep 17 00:00:00 2001 From: yasar-sumologic Date: Fri, 19 Jul 2024 10:22:41 +0530 Subject: [PATCH 1/6] Add metrics collection and metrics-based dashboards info to RabbitMQ Otel --- .../opentelemetry/rabbitmq-opentelemetry.md | 57 +++++++++++++++---- 1 file changed, 47 insertions(+), 10 deletions(-) diff --git a/docs/integrations/containers-orchestration/opentelemetry/rabbitmq-opentelemetry.md b/docs/integrations/containers-orchestration/opentelemetry/rabbitmq-opentelemetry.md index 9761804934..dd469d6201 100644 --- a/docs/integrations/containers-orchestration/opentelemetry/rabbitmq-opentelemetry.md +++ b/docs/integrations/containers-orchestration/opentelemetry/rabbitmq-opentelemetry.md @@ -11,7 +11,7 @@ import TabItem from '@theme/TabItem'; icon Thumbnail icon -The [RabbitMQ](https://www.rabbitmq.com/getstarted.html) app is a unified log app. Preconfigured dashboards provide insight into error logs. RabbitMQ logs are sent to Sumo Logic through OpenTelemetry [filelog receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver). +The [RabbitMQ](https://www.rabbitmq.com/getstarted.html) app is a unified log app. Preconfigured dashboards provide insight into error logs and metrics. RabbitMQ logs and metrics are sent to Sumo Logic through OpenTelemetry [filelog receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver). Schematics @@ -19,17 +19,27 @@ The [RabbitMQ](https://www.rabbitmq.com/getstarted.html) app is a unified log ap Following are the [Fields](/docs/manage/fields/) which will be created as part of RabbitMQ App install if not already present. -* `messaging.cluster.name`. User configured. Specify the user-friendly cluster name which RabbitMQ belongs to. -* `sumo.datasource`. Has fixed value of **rabbitmq**. +* **`sumo.datasource`**. Fixed value of **rabbitmq**. +* **`messaging.system`**. Fixed value of **rabbitmq**. +* **`deployment.environment`**. This is a user-configured field set at the time of collector installation. It identifies the environment where the rabbitmq env resides, such as `dev`, `prod`, or `qa`. +* **`messaging.cluster.name`**. User configured. Enter a name to uniquely identify your Rabbitmq cluster. This cluster name will be shown in the Sumo Logic dashboards. +* **`messaging.node.name`**. Includes the value of the hostname of the machine which is being monitored. ## Prerequisites +### For metrics collection + +This RabbitMQ receiver fetches stats from a RabbitMQ node using the [RabbitMQ Management Plugin](https://www.rabbitmq.com/docs/management) and converts them to OpenTelemetry metrics + +### For logs collection + This section provides instructions for configuring log collection for RabbitMQ running on a non-Kubernetes environment for the Sumo Logic App for RabbitMQ. By default, RabbitMQ logs are stored in a log file. Follow the instructions to set up log collection: -1. **Configure logging in RabbitMQ**. RabbitMQ supports logging via the following methods: local text log files, syslog and stdout. RabbitMQ logs have six levels of verbosity: debug, info, warning, error, critical, none. For details please visit this [page](https://www.rabbitmq.com/logging.html#log-levels). For the dashboards to work properly, log level needs to be set to **debug**. Default log level is **info**. All logging settings are located in [RabbitMQ.conf](https://www.rabbitmq.com/logging.html). -2. **Configure RabbitMQ to write log lines to a local file**. By default, RabbitMQ logs are stored in `/var/log/rabbitmq/rabbit@.log`. The default directory for log files is listed in the `RabbitMQ.conf` file. To configure the log output destination to a log file, use one of the following settings, either in the [configuration file](https://www.rabbitmq.com/logging.html). Edit or create `/etc/rabbitmq/rabbitmq.conf` file config: + +1. RabbitMQ logs have six levels of verbosity: debug, info, warning, error, critical, none. For details please visit this [page](https://www.rabbitmq.com/logging.html#log-levels). For the dashboards to work properly, log level needs to be set to **debug**. Default log level is **info**. All logging settings are located in [RabbitMQ.conf](https://www.rabbitmq.com/logging.html). +2. Configure RabbitMQ to write log lines to a local file. By default, RabbitMQ logs are stored in `/var/log/rabbitmq/rabbit@.log`. The default directory for log files is listed in the `RabbitMQ.conf` file. To configure the log output destination to a log file, use one of the following settings, either in the [configuration file](https://www.rabbitmq.com/logging.html). Edit or create `/etc/rabbitmq/rabbitmq.conf` file config: ``` log.dir = /var/log/rabbitmq log.file = rabbitmq.log @@ -74,9 +84,18 @@ import SetupColl from '../../../reuse/apps/opentelemetry/set-up-collector.md'; ### Step 2: Configure integration -In this step, we will be configuring the yaml file required for RabbitMQ Collection. Path of the log file configured to capture RabbitMQ logs is needed to be given here. +OpenTelemetry works with a [configuration](https://opentelemetry.io/docs/collector/configuration/) yaml file with all the details concerning the data that needs to be collected. For example, it specifies the location of a log file that is read and sent to the Sumo Logic platform. + +In this step, you will configure the yaml file required for RabbitMQ Collection. -The files are typically located in `/var/log/rabbitmq/rabbit@.log`. You can add any custom fields which you want to tag along with the data ingested in sumo. Click on the **Download YAML File** button to get the yaml file. +Below are the inputs required: + +- **`endpoint (no default)`**. The hostname and port of the RabbitMQ instance, separated by a colon. (For example: `localhost:15672`.) +- **RabbitMQ logs Path**. Enter the path to the log file for your RabbitMQ instance. + +The log file path configured to capture rabbitmq logs must be given here. The files are typically located in `/var/log/rabbitmq/rabbit@.log`. If you are using a customized path, check the [`rabbitmq.conf`](https://www.rabbitmq.com/logging.html) file for this information. + +You can add any custom fields which you want to tag along with the data ingested in sumo. Click on the **Download YAML File** button to get the yaml file. YAML @@ -164,6 +183,8 @@ Here's a sample log message you'd find in Non-Kubernetes environments. ## Sample queries +### Logs + This sample Query is from the **RabbitMQ - Logs dashboard** > **Events** by Severity panel. ```sql title="Query String" @@ -173,10 +194,26 @@ This sample Query is from the **RabbitMQ - Logs dashboard** > **Events** by Seve | parse "* * [*]" as date,time,severity | count by severity ``` +### Metrics + +This sample query is from the **RabbitMQ - Metrics dashboard** > **Average Number of Consumers** panel. + +```sql sumo.datasource=rabbitmq metric=rabbitmq.consumer.count deployment.environment=* messaging.cluster.name=* messaging.node.name=* rabbitmq.queue.name=* rabbitmq.vhost.name=* +| avg by messaging.cluster.name +| sum +``` + + ## Viewing RabbitMQ Dashboards -### Overview +### Logs + +The **RabbitMQ - Logs** dashboard gives you an at-a-glance view of Error messages, error by severity, top and last 10 errors, Broker and Event Start/Add log messages. + +RabbitMQ Logs dashboards + +### Metrics -The **RabbitMQ - Overview** dashboard gives you an at-a-glance view of Error messages, error by severity, top and last 10 errors, Broker and Event Start/Add log messages. +The **RabbitMQ - Metrics** dashboard gives you an at-a-glance view of your RabbitMQ deployment across brokers, queue, exchange, consumer, messages. -Overview +RabbitMQ Metrics dashboards From 84dc37d871a75db7182877d26b66c58ae5869ced Mon Sep 17 00:00:00 2001 From: yasar-sumologic Date: Mon, 22 Jul 2024 10:09:50 +0530 Subject: [PATCH 2/6] fixed PR review comments --- .../opentelemetry/rabbitmq-opentelemetry.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/integrations/containers-orchestration/opentelemetry/rabbitmq-opentelemetry.md b/docs/integrations/containers-orchestration/opentelemetry/rabbitmq-opentelemetry.md index dd469d6201..3cb2b6f2de 100644 --- a/docs/integrations/containers-orchestration/opentelemetry/rabbitmq-opentelemetry.md +++ b/docs/integrations/containers-orchestration/opentelemetry/rabbitmq-opentelemetry.md @@ -11,7 +11,7 @@ import TabItem from '@theme/TabItem'; icon Thumbnail icon -The [RabbitMQ](https://www.rabbitmq.com/getstarted.html) app is a unified log app. Preconfigured dashboards provide insight into error logs and metrics. RabbitMQ logs and metrics are sent to Sumo Logic through OpenTelemetry [filelog receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver). +The Sumo Logic app for [RabbitMQ](https://www.rabbitmq.com/getstarted.html) is a unified log solution that includes preconfigured dashboards offering insights into error logs and metrics. RabbitMQ logs and metrics are transmitted to Sumo Logic via the OpenTelemetry [filelog receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver). Schematics @@ -22,14 +22,14 @@ Following are the [Fields](/docs/manage/fields/) which will be created as part o * **`sumo.datasource`**. Fixed value of **rabbitmq**. * **`messaging.system`**. Fixed value of **rabbitmq**. * **`deployment.environment`**. This is a user-configured field set at the time of collector installation. It identifies the environment where the rabbitmq env resides, such as `dev`, `prod`, or `qa`. -* **`messaging.cluster.name`**. User configured. Enter a name to uniquely identify your Rabbitmq cluster. This cluster name will be shown in the Sumo Logic dashboards. +* **`messaging.cluster.name`**. User configured. Enter a name to uniquely identify your RabbitMQ cluster. This cluster name will be shown in the Sumo Logic dashboards. * **`messaging.node.name`**. Includes the value of the hostname of the machine which is being monitored. ## Prerequisites ### For metrics collection -This RabbitMQ receiver fetches stats from a RabbitMQ node using the [RabbitMQ Management Plugin](https://www.rabbitmq.com/docs/management) and converts them to OpenTelemetry metrics +This RabbitMQ receiver fetches stats from a RabbitMQ node using the [RabbitMQ Management Plugin](https://www.rabbitmq.com/docs/management) and converts them to OpenTelemetry metrics. ### For logs collection @@ -38,7 +38,7 @@ This section provides instructions for configuring log collection for RabbitMQ r Follow the instructions to set up log collection: -1. RabbitMQ logs have six levels of verbosity: debug, info, warning, error, critical, none. For details please visit this [page](https://www.rabbitmq.com/logging.html#log-levels). For the dashboards to work properly, log level needs to be set to **debug**. Default log level is **info**. All logging settings are located in [RabbitMQ.conf](https://www.rabbitmq.com/logging.html). +1. RabbitMQ logs have six levels of verbosity: debug, info, warning, error, critical, and none. For details, visit the [RabbitMQ documentation](https://www.rabbitmq.com/logging.html#log-levels). For the dashboards to work properly, the log level needs to be set to **debug**. Default log level is **info**. All logging settings are located in [RabbitMQ.conf](https://www.rabbitmq.com/logging.html). 2. Configure RabbitMQ to write log lines to a local file. By default, RabbitMQ logs are stored in `/var/log/rabbitmq/rabbit@.log`. The default directory for log files is listed in the `RabbitMQ.conf` file. To configure the log output destination to a log file, use one of the following settings, either in the [configuration file](https://www.rabbitmq.com/logging.html). Edit or create `/etc/rabbitmq/rabbitmq.conf` file config: ``` log.dir = /var/log/rabbitmq @@ -93,7 +93,7 @@ Below are the inputs required: - **`endpoint (no default)`**. The hostname and port of the RabbitMQ instance, separated by a colon. (For example: `localhost:15672`.) - **RabbitMQ logs Path**. Enter the path to the log file for your RabbitMQ instance. -The log file path configured to capture rabbitmq logs must be given here. The files are typically located in `/var/log/rabbitmq/rabbit@.log`. If you are using a customized path, check the [`rabbitmq.conf`](https://www.rabbitmq.com/logging.html) file for this information. +The log file path configured to capture RabbitMQ logs must be given here. The files are typically located in `/var/log/rabbitmq/rabbit@.log`. If you are using a customized path, check the [`rabbitmq.conf`](https://www.rabbitmq.com/logging.html) file for this information. You can add any custom fields which you want to tag along with the data ingested in sumo. Click on the **Download YAML File** button to get the yaml file. From 1e5921a1f724eb39a8f80aed58553f6de7973582 Mon Sep 17 00:00:00 2001 From: yasar-sumologic Date: Mon, 22 Jul 2024 12:50:33 +0530 Subject: [PATCH 3/6] fixed PR review comments --- .../opentelemetry/rabbitmq-opentelemetry.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/docs/integrations/containers-orchestration/opentelemetry/rabbitmq-opentelemetry.md b/docs/integrations/containers-orchestration/opentelemetry/rabbitmq-opentelemetry.md index 3cb2b6f2de..ba4b6e6781 100644 --- a/docs/integrations/containers-orchestration/opentelemetry/rabbitmq-opentelemetry.md +++ b/docs/integrations/containers-orchestration/opentelemetry/rabbitmq-opentelemetry.md @@ -11,7 +11,7 @@ import TabItem from '@theme/TabItem'; icon Thumbnail icon -The Sumo Logic app for [RabbitMQ](https://www.rabbitmq.com/getstarted.html) is a unified log solution that includes preconfigured dashboards offering insights into error logs and metrics. RabbitMQ logs and metrics are transmitted to Sumo Logic via the OpenTelemetry [filelog receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver). +RabbitMQ logs are sent to Sumo Logic through the OpenTelemetry [filelog receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver). RabbitMQ metrics are sent through the [RabbitMQ](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/rabbitmqreceiver) metrics receiver. Schematics @@ -29,7 +29,10 @@ Following are the [Fields](/docs/manage/fields/) which will be created as part o ### For metrics collection -This RabbitMQ receiver fetches stats from a RabbitMQ node using the [RabbitMQ Management Plugin](https://www.rabbitmq.com/docs/management) and converts them to OpenTelemetry metrics. +The RabbitMQ receiver supports RabbitMQ versions `3.8` and `3.9`. +The RabbitMQ Management Plugin must be enabled by following the [official instructions](https://www.rabbitmq.com/management.html#getting-started). + +Also, a user with at least [monitoring](https://www.rabbitmq.com/management.html#permissions) level permissions must be used for monitoring. ### For logs collection @@ -92,11 +95,15 @@ Below are the inputs required: - **`endpoint (no default)`**. The hostname and port of the RabbitMQ instance, separated by a colon. (For example: `localhost:15672`.) - **RabbitMQ logs Path**. Enter the path to the log file for your RabbitMQ instance. +- **username**. enter the RabbitMQ username. +- **password**. enter the RabbitMQ password. The log file path configured to capture RabbitMQ logs must be given here. The files are typically located in `/var/log/rabbitmq/rabbit@.log`. If you are using a customized path, check the [`rabbitmq.conf`](https://www.rabbitmq.com/logging.html) file for this information. You can add any custom fields which you want to tag along with the data ingested in sumo. Click on the **Download YAML File** button to get the yaml file. +For Linux platform, click **Download Environment Variables File** to get the file with the password which is supposed to be set as environment variable. + YAML ### Step 3: Send logs to Sumo Logic From 7cc71731fd7cb9b69809722526876b6dedbca114 Mon Sep 17 00:00:00 2001 From: yasar-sumologic Date: Mon, 22 Jul 2024 13:14:31 +0530 Subject: [PATCH 4/6] Update rabbitmq-opentelemetry.md --- .../opentelemetry/rabbitmq-opentelemetry.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/integrations/containers-orchestration/opentelemetry/rabbitmq-opentelemetry.md b/docs/integrations/containers-orchestration/opentelemetry/rabbitmq-opentelemetry.md index ba4b6e6781..5d9efd1d49 100644 --- a/docs/integrations/containers-orchestration/opentelemetry/rabbitmq-opentelemetry.md +++ b/docs/integrations/containers-orchestration/opentelemetry/rabbitmq-opentelemetry.md @@ -95,8 +95,8 @@ Below are the inputs required: - **`endpoint (no default)`**. The hostname and port of the RabbitMQ instance, separated by a colon. (For example: `localhost:15672`.) - **RabbitMQ logs Path**. Enter the path to the log file for your RabbitMQ instance. -- **username**. enter the RabbitMQ username. -- **password**. enter the RabbitMQ password. +- **username**. Enter the RabbitMQ username. +- **password**. Enter the RabbitMQ password. The log file path configured to capture RabbitMQ logs must be given here. The files are typically located in `/var/log/rabbitmq/rabbit@.log`. If you are using a customized path, check the [`rabbitmq.conf`](https://www.rabbitmq.com/logging.html) file for this information. From ddd5b2d2191ece7a651b2e26e3d0e08d29d9cabb Mon Sep 17 00:00:00 2001 From: YasarArafat Nasir Rogangar <163980163+yasar-sumologic@users.noreply.github.com> Date: Wed, 24 Jul 2024 11:28:09 +0530 Subject: [PATCH 5/6] Apply suggestions from code review Co-authored-by: Jagadisha V <129049263+JV0812@users.noreply.github.com> --- .../opentelemetry/rabbitmq-opentelemetry.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/integrations/containers-orchestration/opentelemetry/rabbitmq-opentelemetry.md b/docs/integrations/containers-orchestration/opentelemetry/rabbitmq-opentelemetry.md index 5d9efd1d49..516b9e709c 100644 --- a/docs/integrations/containers-orchestration/opentelemetry/rabbitmq-opentelemetry.md +++ b/docs/integrations/containers-orchestration/opentelemetry/rabbitmq-opentelemetry.md @@ -29,10 +29,12 @@ Following are the [Fields](/docs/manage/fields/) which will be created as part o ### For metrics collection +:::note The RabbitMQ receiver supports RabbitMQ versions `3.8` and `3.9`. -The RabbitMQ Management Plugin must be enabled by following the [official instructions](https://www.rabbitmq.com/management.html#getting-started). +::: + +Enable the RabbitMQ management plugin by following the instructions in the [RabbitMQ documentation](https://www.rabbitmq.com/management.html#getting-started). Also, make sure you at least have the [monitoring](https://www.rabbitmq.com/management.html#permissions) level permissions to monitor the metrics. -Also, a user with at least [monitoring](https://www.rabbitmq.com/management.html#permissions) level permissions must be used for monitoring. ### For logs collection @@ -41,7 +43,7 @@ This section provides instructions for configuring log collection for RabbitMQ r Follow the instructions to set up log collection: -1. RabbitMQ logs have six levels of verbosity: debug, info, warning, error, critical, and none. For details, visit the [RabbitMQ documentation](https://www.rabbitmq.com/logging.html#log-levels). For the dashboards to work properly, the log level needs to be set to **debug**. Default log level is **info**. All logging settings are located in [RabbitMQ.conf](https://www.rabbitmq.com/logging.html). +1. RabbitMQ logs have six levels of verbosity: debug, info, warning, error, critical, and none. For more information, refer to the [RabbitMQ documentation](https://www.rabbitmq.com/logging.html#log-levels). To make sure the dashboard work properly, the log level needs to be set to **debug**. Default log level is **info**. All logging settings are located in [RabbitMQ.conf](https://www.rabbitmq.com/logging.html). 2. Configure RabbitMQ to write log lines to a local file. By default, RabbitMQ logs are stored in `/var/log/rabbitmq/rabbit@.log`. The default directory for log files is listed in the `RabbitMQ.conf` file. To configure the log output destination to a log file, use one of the following settings, either in the [configuration file](https://www.rabbitmq.com/logging.html). Edit or create `/etc/rabbitmq/rabbitmq.conf` file config: ``` log.dir = /var/log/rabbitmq @@ -89,7 +91,7 @@ import SetupColl from '../../../reuse/apps/opentelemetry/set-up-collector.md'; OpenTelemetry works with a [configuration](https://opentelemetry.io/docs/collector/configuration/) yaml file with all the details concerning the data that needs to be collected. For example, it specifies the location of a log file that is read and sent to the Sumo Logic platform. -In this step, you will configure the yaml file required for RabbitMQ Collection. +In this step, you will configure the yaml file required for RabbitMQ collection. Below are the inputs required: From b582baa817db634ee8c26efa2686e2a2a59b0787 Mon Sep 17 00:00:00 2001 From: YasarArafat Nasir Rogangar <163980163+yasar-sumologic@users.noreply.github.com> Date: Wed, 24 Jul 2024 11:49:50 +0530 Subject: [PATCH 6/6] Apply suggestions from code review Co-authored-by: Jagadisha V <129049263+JV0812@users.noreply.github.com> --- .../opentelemetry/rabbitmq-opentelemetry.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/integrations/containers-orchestration/opentelemetry/rabbitmq-opentelemetry.md b/docs/integrations/containers-orchestration/opentelemetry/rabbitmq-opentelemetry.md index 516b9e709c..9e5152b4b1 100644 --- a/docs/integrations/containers-orchestration/opentelemetry/rabbitmq-opentelemetry.md +++ b/docs/integrations/containers-orchestration/opentelemetry/rabbitmq-opentelemetry.md @@ -205,7 +205,7 @@ This sample Query is from the **RabbitMQ - Logs dashboard** > **Events** by Seve ### Metrics -This sample query is from the **RabbitMQ - Metrics dashboard** > **Average Number of Consumers** panel. +This sample query is from the **Average Number of Consumers** panel in **RabbitMQ - Metrics** dashboard. ```sql sumo.datasource=rabbitmq metric=rabbitmq.consumer.count deployment.environment=* messaging.cluster.name=* messaging.node.name=* rabbitmq.queue.name=* rabbitmq.vhost.name=* | avg by messaging.cluster.name @@ -217,12 +217,12 @@ This sample query is from the **RabbitMQ - Metrics dashboard** > **Average Numbe ### Logs -The **RabbitMQ - Logs** dashboard gives you an at-a-glance view of Error messages, error by severity, top and last 10 errors, Broker and Event Start/Add log messages. +The **RabbitMQ - Logs** dashboard gives you an at-a-glance view of error messages, error by severity, top and last 10 errors, broker, and Event Start/Add log messages. RabbitMQ Logs dashboards ### Metrics -The **RabbitMQ - Metrics** dashboard gives you an at-a-glance view of your RabbitMQ deployment across brokers, queue, exchange, consumer, messages. +The **RabbitMQ - Metrics** dashboard gives you an at-a-glance view of your RabbitMQ deployment across brokers, queue, exchange, consumer, and messages. RabbitMQ Metrics dashboards