Skip to content

Commit c81feeb

Browse files
yasar-sumologicsumoanemaJV0812jpipkin1
authored
Added metrics collection and metrics-based dashboards info to MongoDB (#4517)
* Added metrics collection and metrics-based dashboards info to MongoDB * fixed it. * Update mongodb-opentelemetry.md * fixed PR review comments * Added Linux env download description * Update haproxy-opentelemetry.md * Update redis-opentelemetry.md * Update rabbitmq-opentelemetry.md * Apply suggestions from code review Co-authored-by: Jagadisha V <129049263+JV0812@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Jagadisha V <129049263+JV0812@users.noreply.github.com> * Update mongodb-opentelemetry.md * fixed PR review comment * Update docs/integrations/databases/opentelemetry/redis-opentelemetry.md * Update docs/integrations/databases/opentelemetry/mongodb-opentelemetry.md * Update docs/integrations/databases/opentelemetry/mongodb-opentelemetry.md * Update mongodb-opentelemetry.md * Fix build break * Updates for review --------- Co-authored-by: Alekh Nema <91047769+sumoanema@users.noreply.github.com> Co-authored-by: Jagadisha V <129049263+JV0812@users.noreply.github.com> Co-authored-by: John Pipkin <jpipkin@sumologic.com>
1 parent a2c5c45 commit c81feeb

File tree

3 files changed

+119
-49
lines changed

3 files changed

+119
-49
lines changed

docs/integrations/containers-orchestration/opentelemetry/rabbitmq-opentelemetry.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -129,7 +129,11 @@ import LogsIntro from '../../../reuse/apps/opentelemetry/send-logs-intro.md';
129129
<TabItem value="Linux">
130130

131131
1. Copy the yaml file to `/etc/otelcol-sumo/conf.d/` folder in the RabbitMQ instance which needs to be monitored.
132-
2. Restart the collector using:
132+
2. Move the `env` file to the following directory:
133+
```sh
134+
/etc/otelcol-sumo/env/
135+
```
136+
3. Restart the collector using:
133137
```sh
134138
  sudo systemctl restart otelcol-sumo
135139
```

docs/integrations/databases/opentelemetry/mongodb-opentelemetry.md

Lines changed: 94 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -11,15 +11,17 @@ import TabItem from '@theme/TabItem';
1111

1212
<img src={useBaseUrl('img/integrations/databases/mongodb.png')} alt="Thumbnail icon" width="120"/><img src={useBaseUrl('img/send-data/otel-color.svg')} alt="Thumbnail icon" width="45"/>
1313

14-
[MongoDB](https://www.mongodb.com/why-use-mongodb#:~:text=MongoDB%20is%20a%20document%20database,development%20teams%20using%20agile%20methodologies.) is a source-available cross-platform document-oriented database program. The Sumo Logic app for MongoDB supports logs and metrics from the open source version of MongoDB. The App is tested on the 4.4.4 version of MongoDB.
14+
[MongoDB](https://www.mongodb.com/why-use-mongodb#:~:text=MongoDB%20is%20a%20document%20database,development%20teams%20using%20agile%20methodologies.) is a source-available cross-platform document-oriented database program. The Sumo Logic app for MongoDB supports logs and metrics from the open source version of MongoDB. The app is tested on the 7.0.14 version of MongoDB.
1515

16-
MongoDB logs are sent to Sumo Logic through OpenTelemetry [filelog receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver).
16+
MongoDB logs are sent to Sumo Logic through OpenTelemetry [filelog receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver). The OpenTelemetry collector runs on the same host as MongoDB and uses the [MongoDB Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/mongodbreceiver) and the [Sumo Logic OpenTelemetry Exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/sumologicexporter) to send the metrics to Sumo Logic.
1717

1818
<img src='https://sumologic-app-data-v2.s3.amazonaws.com/dashboards/MongoDB-OpenTelemetry/MongoDB-Schematics.png' alt="Schematics" />
1919

20-
## Log types
20+
## Log and metrics types
2121

22-
The MongoDB logs are generated in files as configured in the configuration file `/var/log/mongodb/mongodb.log`. For more details on MongoDB logs, see [this](https://docs.mongodb.com/manual/reference/log-messages/) link.
22+
This app supports logs and metrics for MongoDB instance. The MongoDB logs are generated in files as configured in this configuration file `/var/log/mongodb/mongodb.log`. For more details on MongoDB logs, refer to the [MongoDB documentation](https://docs.mongodb.com/manual/reference/log-messages/).
23+
24+
The app supports metrics generated by the [MongoDB Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/mongodbreceiver/documentation.md).
2325

2426
## Fields creation in Sumo Logic for MongoDB
2527

@@ -29,19 +31,28 @@ Following are the [Fields](/docs/manage/fields/) which will be created as part o
2931
- **`db.system`**. Has fixed value of **mongodb**.
3032
- **`deployment.environment`**. User configured. This is the deployment environment where the Mongodb cluster resides. For example: dev, prod or qa.
3133
- **`sumo.datasource`**. has a fixed value of **mongodb**.
34+
* **`db.node.name`**. Has the value of host name of the machine which is being monitored.
3235

3336
## Prerequisites
3437

38+
### For metrics collection
39+
40+
The MongoDB receiver fetches stats from a MongoDB instance using the [golang mongo driver](https://github.com/mongodb/mongo-go-driver). Stats are collected via MongoDB's `dbStats` and `serverStatus` commands. The MongoDB receiver extracts values from the result and converts them to OpenTelemetry metrics.
41+
42+
### For logs collection
43+
3544
By default, MongoDB logs are stored in a log file.
3645

37-
1. Configure logging verbosity in MongoDB : MongoDB logs have six levels of verbosity. All logging settings are located in [MongoDB.conf](https://docs.mongodb.com/manual/reference/method/db.setLogLevel/). To select a level, set loglevel to one of:
46+
1. Configure logging verbosity in MongoDB.<br/>MongoDB logs have six levels of verbosity. All logging settings are located in [MongoDB.conf](https://docs.mongodb.com/manual/reference/method/db.setLogLevel/). To select a level, set loglevel to one of:
3847
- 0 is the MongoDB's default log verbosity level, to include [Informational](https://docs.mongodb.com/manual/reference/log-messages/#std-label-log-severity-levels) messages.
3948
- 1 to 5 increases the verbosity level to include[ Debug](https://docs.mongodb.com/manual/reference/log-messages/#std-label-log-severity-levels) messages.
40-
2. Configure MongoDB to log to a Local file: Configuring MongoDB logs to go to log files. By default, MongoDB logs are stored in `/var/log/mongodb/mongodb.log`. The default directory for log files is listed in the MongoDB.conf file. To configure the log output destination to a log file, use one of the following settings, either in the [configuration file](https://docs.mongodb.com/manual/reference/configuration-options/) or command-line:
49+
2. You need to set the [profiling_level](https://www.mongodb.com/docs/manual/reference/method/db.setProfilingLevel/) to a value of 1. <br/>But by default it is set to 0. You can also configure the value of [slowms](https://www.mongodb.com/docs/manual/reference/method/db.setProfilingLevel/#std-label-set-profiling-level-options-slowms) to a value greater than or equal to 100. Thus can be done using the below command in mongosh:
50+
`db.setProfilingLevel(1,100)`
51+
3. Configure MongoDB to log to a local file. <br/>By default, MongoDB logs are stored in `/var/log/mongodb/mongodb.log`. The default directory for log files is listed in the `MongoDB.conf` file. To configure the log output destination to a log file, use one of the following settings, either in the [configuration file](https://docs.mongodb.com/manual/reference/configuration-options/) or command-line:
4152
- Configuration file: The [systemLog.destination](https://docs.mongodb.com/manual/reference/configuration-options/#mongodb-setting-systemLog.destination) option for file.
42-
- Command-line:
43-
- The [--logpath](https://docs.mongodb.com/manual/reference/program/mongod/#std-option-mongod.--logpath) option for [mongod](https://docs.mongodb.com/manual/reference/program/mongod/#mongodb-binary-bin.mongod) for file.
44-
- The [--logpath](https://docs.mongodb.com/manual/reference/program/mongos/#std-option-mongos.--logpath) option for [mongos](https://docs.mongodb.com/manual/reference/program/mongos/#mongodb-binary-bin.mongos) for file.
53+
- Command-line:
54+
- The [--logpath](https://docs.mongodb.com/manual/reference/program/mongod/#std-option-mongod.--logpath) option for [mongod](https://docs.mongodb.com/manual/reference/program/mongod/#mongodb-binary-bin.mongod) for file.
55+
- The [--logpath](https://docs.mongodb.com/manual/reference/program/mongos/#std-option-mongos.--logpath) option for [mongos](https://docs.mongodb.com/manual/reference/program/mongos/#mongodb-binary-bin.mongos) for file.
4556

4657
import LogsCollectionPrereqisites from '../../../reuse/apps/logs-collection-prereqisites.md';
4758

@@ -79,9 +90,22 @@ import SetupColl from '../../../reuse/apps/opentelemetry/set-up-collector.md';
7990

8091
### Step 2: Configure integration
8192

82-
In this step the user needs to provide the path to the mongo db log file configured as part of above steps. Typically the logs are located at the location: `/var/log/mongodb/mongodb.log`.
93+
OpenTelemetry works with a [configuration](https://opentelemetry.io/docs/collector/configuration/) YAML file with all the details concerning the data that needs to be collected. For example, it specifies the location of a log file that is read and sent to the Sumo Logic platform.
94+
95+
In this step, you will configure the YAML file required for MongoDB collection.
96+
97+
Below are the inputs required:
98+
99+
- **`Endpoint (no default)`**. The hostname and port of the MognoDB instance, separated by a colon. (For example: `localhost:27017`.)
100+
- **`Logs Path`**. Provide the path to the mongo db log file configured as part of above steps. Typically the logs are located at the location: `/var/log/mongodb/mongodb.log`.
101+
- **`username (optional)`**. If authentication is required, the user can with clusterMonitor permissions can be provided here.
102+
- **`password (optional)`**. If authentication is required, the password can be provided here.
103+
104+
You can add any custom fields which you want to tag along with the data ingested in Sumo. Click on the **Download YAML File** button to get the YAML file.
83105

84-
You can add any custom fields which you want to tag along with the data ingested in Sumo. Click on the **Download YAML File** button to get the yaml file.
106+
Click on the **Download YAML File** button to get the YAML file.
107+
108+
For Linux platform, click **Download Environment Variables File** to get the file with the password which is supposed to be set as environment variable.
85109

86110
<img src='https://sumologic-app-data-v2.s3.amazonaws.com/dashboards/MongoDB-OpenTelemetry/MongoDB-YAML.png' style={{border:'1px solid gray'}} alt="YAML" />
87111

@@ -105,16 +129,20 @@ import LogsIntro from '../../../reuse/apps/opentelemetry/send-logs-intro.md';
105129

106130
<TabItem value="Linux">
107131

108-
1. Copy the yaml file to `/etc/otelcol-sumo/conf.d/` folder in the Mongodb instance which needs to be monitored.
109-
2. restart the collector using:
132+
1. Copy the YAML file to `/etc/otelcol-sumo/conf.d/` folder in the Mongodb instance which needs to be monitored.
133+
2. Place Env file in the following directory:
134+
```sh
135+
/etc/otelcol-sumo/env/
136+
```
137+
3. restart the collector using:
110138
```sh
111139
 sudo systemctl restart otelcol-sumo
112140
```
113141

114142
</TabItem>
115143
<TabItem value="Windows">
116144

117-
1. Copy the yaml file to `C:\ProgramData\Sumo Logic\OpenTelemetry Collector\config\conf.d` folder in the machine which needs to be monitored.
145+
1. Copy the YAML file to `C:\ProgramData\Sumo Logic\OpenTelemetry Collector\config\conf.d` folder in the machine which needs to be monitored.
118146
2. Restart the collector using: 
119147
```sh
120148
Restart-Service -Name OtelcolSumo
@@ -123,7 +151,7 @@ Restart-Service -Name OtelcolSumo
123151
</TabItem>
124152
<TabItem value="macOS">
125153

126-
1. Copy the yaml file to `/etc/otelcol-sumo/conf.d/` folder in the Mongodb instance which needs to be monitored.
154+
1. Copy the YAML file to `/etc/otelcol-sumo/conf.d/` folder in the Mongodb instance which needs to be monitored.
127155
2. Restart the otelcol-sumo process using:
128156
```sh
129157
 otelcol-sumo --config /etc/otelcol-sumo/sumologic.yaml --config "glob:/etc/otelcol-sumo/conf.d/*.yaml" 
@@ -161,7 +189,7 @@ import LogsOutro from '../../../reuse/apps/opentelemetry/send-logs-outro.md';
161189

162190
## Sample log messages
163191

164-
```sql
192+
```json
165193
{
166194
   "t":{
167195
       "$date":"2021-05-21T10:22:57.373+00:00"
@@ -195,9 +223,7 @@ import LogsOutro from '../../../reuse/apps/opentelemetry/send-logs-outro.md';
195223

196224
## Sample queries
197225

198-
Dashboard: MongoDB - Errors and Warnings, Panel: Errors by Component
199-
200-
```sql
226+
```sql title="Errors by Component( MongoDB - Errors and Warnings)"
201227
deployment.environment=* db.cluster.name=* sumo.datasource=mongodb  | json "log" as _rawlog nodrop
202228
| if (isEmpty(_rawlog), _raw, _rawlog) as _raw
203229
| json field=_raw "t.$date" as timestamp
@@ -211,17 +237,19 @@ deployment.environment=* db.cluster.name=* sumo.datasource=mongodb  | json "log
211237

212238
## Viewing MongoDB dashboards
213239

214-
If no relevant data was received within the time range of the Panel, the Panel will be empty.
240+
import ViewDashboards from '../../../reuse/apps/view-dashboards.md';
241+
242+
<ViewDashboards/>
215243

216244
### Overview
217245

218-
The **MongoDB - Overview** dashboard provides an at-a-glance view of MongoDB health, performance and problems causing errors.
246+
The **MongoDB - Overview** dashboard provides an at-a-glance view of MongoDB health, performance, and problems causing errors.
219247

220248
Use this dashboard to:
221249

222-
- Identify Slow CRUD and DB commands.
223-
- Gain insights into Errors logs by component and context.
224-
- Number of up servers.
250+
- Identify slow CRUD and DB commands.
251+
- Gain insights into errors logs by component and context.
252+
- Know the number of up servers.
225253

226254
<img src='https://sumologic-app-data-v2.s3.amazonaws.com/dashboards/MongoDB-OpenTelemetry/MongoDB-Overview.png' alt="Overview" />
227255

@@ -231,7 +259,7 @@ The **MongoDB - Errors and Warnings** dashboard shows errors and warnings by the
231259

232260
Use this dashboard to:
233261

234-
- Determine components producing multiple errors or warnings.
262+
- Determine components producing multiple errors or warnings.
235263

236264
<img src='https://sumologic-app-data-v2.s3.amazonaws.com/dashboards/MongoDB-OpenTelemetry/MongoDB-Errors-and-Warnings.png' alt="Access" />
237265

@@ -241,8 +269,8 @@ The **MongoDB - Logins and Connections** dashboard shows geo location of client
241269

242270
Use this dashboard to:
243271

244-
- Determine potential hacking attempts.
245-
- Determine location of attacks.
272+
- Determine potential hacking attempts.
273+
- Determine location of attacks.
246274

247275
<img src='https://sumologic-app-data-v2.s3.amazonaws.com/dashboards/MongoDB-OpenTelemetry/MongoDB-Logins-and-Connections.png' alt="Logins and Connections" />
248276

@@ -252,29 +280,59 @@ The **MongoDB - Query Logs** dashboard shows read and write query trends.
252280

253281
Use this dashboard to:
254282

255-
- Monitor abnormal spikes in Query volume.
256-
- Identify the read versus write ratio of your application queries. Adjusting indexes to improve query performance.
283+
- Monitor abnormal spikes in query volume.
284+
- Identify read versus write ratio of your application queries. This helps you to adjust indexes to improve query performance.
257285

258286
<img src='https://sumologic-app-data-v2.s3.amazonaws.com/dashboards/MongoDB-OpenTelemetry/MongoDB-Query-Logs.png' alt="Query Logs" />
259287

288+
### Sharding
289+
290+
The **MongoDB - Sharding** dashboard shows sharding related errors, events, failures, and number of chunks moving between shards.
291+
292+
Use this dashboard to:
293+
294+
- Identify sharding errors and warnings.
295+
- Gain insights into chunk operations.
296+
297+
<img src='https://sumologic-app-data-v2.s3.amazonaws.com/dashboards/MongoDB-OpenTelemetry/MongoDB-Sharding.png' alt="Sharding" />
298+
260299
### Replication Logs
261300

262301
The **MongoDB - Replication Logs** dashboard shows replica deletes/updates/inserts trend and replica state.
263302

264303
Use this dashboard to:
265304

266-
- Monitor replication state and replication events like inserts/updates/commands per second.
267-
- Track Replication Oplog window to identify replication delay.
305+
- Identify replication errors and warnings.
306+
- Gain insights into replication operations.
268307

269308
<img src='https://sumologic-app-data-v2.s3.amazonaws.com/dashboards/MongoDB-OpenTelemetry/MongoDB-Replication-Logs.png' alt="Replication Logs" />
270309

271-
### Sharding
310+
### Resource
272311

273-
The **MongoDB - Sharding** dashboard dashboard shows sharding related errors, events, failures and number of chunks moving between shards.
312+
The **MongoDB - Resource** dashboard shows resource utilization by the MongoDB component.
274313

275314
Use this dashboard to:
315+
* Determine memory and disk usage.
316+
* Identify potential resource constraints and issues.
276317

277-
- Identify Sharding errors and warnings.
278-
- Gain insights into Chunk operations.
318+
<img src='https://sumologic-app-data-v2.s3.amazonaws.com/dashboards/MongoDB-OpenTelemetry/MongoDB-Resource.png' alt="Resource" />
279319

280-
<img src='https://sumologic-app-data-v2.s3.amazonaws.com/dashboards/MongoDB-OpenTelemetry/MongoDB-Sharding.png' alt="Sharding" />
320+
### Operations
321+
322+
The **MongoDB - Operations** dashboard shows MongoDB queries analytics using metrics.
323+
324+
Use this dashboard to:
325+
* Know different kind of operation count like query, insert, and delete.
326+
* Determine the operation time taken by different queries.
327+
328+
<img src='https://sumologic-app-data-v2.s3.amazonaws.com/dashboards/MongoDB-OpenTelemetry/MongoDB-Operations.png' alt="Operations" />
329+
330+
### Replication
331+
332+
The **MongoDB - Replication** dashboard displays the replication events, errors, warnings, and nodes information.
333+
334+
Use this dashboard to:
335+
* Know different kind of operation count executed on replicas.
336+
* Operation count like query, insert, and delete.
337+
338+
<img src='https://sumologic-app-data-v2.s3.amazonaws.com/dashboards/MongoDB-OpenTelemetry/MongoDB-Replication.png' alt="Replication" />

docs/integrations/databases/opentelemetry/redis-opentelemetry.md

Lines changed: 20 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -75,9 +75,9 @@ import SetupColl from '../../../reuse/apps/opentelemetry/set-up-collector.md';
7575

7676
### Step 2: Configure integration
7777

78-
OpenTelemetry works with a [configuration](https://opentelemetry.io/docs/collector/configuration/) yaml file with all the details concerning the data that needs to be collected. For example, it specifies the location of a log file that is read and sent to the Sumo Logic platform.
78+
OpenTelemetry works with a [configuration](https://opentelemetry.io/docs/collector/configuration/) YAML file with all the details concerning the data that needs to be collected. For example, it specifies the location of a log file that is read and sent to the Sumo Logic platform.
7979

80-
In this step, you will configure the yaml file required for Redis Collection.
80+
In this step, you will configure the YAML file required for Redis Collection.
8181

8282
Below are the inputs required:
8383

@@ -86,7 +86,11 @@ Below are the inputs required:
8686

8787
The log file path configured to capture redis logs must be given here. The files are typically located in `/var/log/redis/redis-server.log`. If you are using a customized path, check the [`redis.conf`](https://download.redis.io/redis-stable/redis.conf) file for this information.
8888

89-
You can add any custom fields which you want to tag along with the data ingested in Sumo. Click on the **Download YAML File** button to get the yaml file.
89+
You can add any custom fields which you want to tag along with the data ingested in Sumo. Click on the **Download YAML File** button to get the YAML file.
90+
91+
Click on the **Download YAML File** button to get the YAML file.
92+
93+
For Linux platform, click **Download Environment Variables File** to get the file with the password which is supposed to be set as environment variable.
9094

9195
<img src='https://sumologic-app-data-v2.s3.amazonaws.com/dashboards/Redis-OpenTelemetry/Redis-YAML.png' style={{border:'1px solid gray'}} alt="Configuration" />
9296

@@ -113,20 +117,24 @@ import LogsIntro from '../../../reuse/apps/opentelemetry/send-logs-intro.md';
113117

114118
<TabItem value="Linux">
115119

116-
1. Copy the yaml at `/etc/otelcol-sumo/conf.d/` folder in the Redis instance that needs to be monitored.
117-
2. Restart the otelcol-sumo process using:
118-
```sh
119-
sudo systemctl restart otelcol-sumo
120-
```
120+
1. Copy the YAML at `/etc/otelcol-sumo/conf.d/` folder in the Redis instance that needs to be monitored.
121+
2. Move the env file in the following directory:
122+
```sh
123+
/etc/otelcol-sumo/env/
124+
```
125+
3. Restart the otelcol-sumo process using:
126+
```sh
127+
sudo systemctl restart otelcol-sumo
128+
```
121129

122130
</TabItem>
123131
<TabItem value="macOS">
124132

125-
1. Copy the yaml at `/etc/otelcol-sumo/conf.d/` folder in the Redis instance that needs to be monitored.
133+
1. Copy the YAML at `/etc/otelcol-sumo/conf.d/` folder in the Redis instance that needs to be monitored.
126134
2. Restart the otelcol-sumo process using:
127-
```sh
128-
otelcol-sumo --config /etc/otelcol-sumo/sumologic.yaml --conf "glob:/etc/otelcol-sumo/conf.d/*.yaml"
129-
```
135+
```sh
136+
otelcol-sumo --config /etc/otelcol-sumo/sumologic.yaml --conf "glob:/etc/otelcol-sumo/conf.d/*.yaml"
137+
```
130138

131139
</TabItem>
132140
<TabItem value="Chef">

0 commit comments

Comments
 (0)