mplotqueries

mplotqueries is a tool to visualize operations in MongoDB logfiles. It has several different plot types and can group information in various ways.

Usage

mplotqueries [-h] [--version] logfile [logfile ...]
             [--group GROUP]
             [--logscale]
             [--type {nscanned/n,rsstate,connchurn,histogram,range,scatter,event} ]
             [--overlay [ {add,list,reset} ]]
             [additional plot type parameters]

mplotqueries can also be used with shell pipe syntax:

mlogfilter logfile [parameters] | mplotqueries --type histogram

General Parameters

Help

-h, --help
shows the help text and exits.

Version

--version
shows the version number and exits.

Groupings

Group By

--group GROUP
The group parameter specifies what the data should be grouped on. Grouping can have different meaning for the various plots below, but generally, groups are represented by color. A scatter plot would choose one color for each group. A histogram plot would also choose a color per group, but additionally stack the histogrammed groups on top of each other. Some plots don't support grouping at all. See the plot types below for information about their grouping behavior.

The following values are possible for GROUP for most plots (some plots may not support all groups)

• namespace (default for single file)
• filename (default for multiple files)
• operation (queries, inserts, updates, ...)
• thread
• log2code (not supported by every plot type)
• custom grouping with regular expressions (see Python's regex syntax)

Example

mplotqueries mongod.log --group operation

This command creates a scatter plot on duration (by default) and colors the operations (queries, inserts, updates, deletes, commands, getmores) in individual colors.

Example

mlogfilter mongod.log --operation update --namespace test.users |
  mplotqueries --type histogram --group 'query: { _id: "([^"]*)" }'

This command combination creates a histogram plot on duration of all the update operations on the test.users collection and groups the updates based on the _id object id (extracted by the regular expression). If parentheses are present in the regular expression, then only the first matched group is being used as the group string. If parentheses are not present, the full regex match is being used as group string.

If the number of groups is large, like in this example, it can be reduced with the --group-limit option below.

###### Group Limits `--group-limit N`

This parameter will limit the number of groups to the top N, based on the number of matching lines per group (descending). The remaining groups are then grouped together in a single bucket called other. This option is useful if the number of groups is very large, as repetitions in color (there are only 14 distinct colors) could otherwise make it hard to distinguish all the groups for some plot types.

Example

mplotqueries mongod.log --type range --group log2code --group-limit 10

This command creates a range plot, grouped on log2code, but only displays the 10 most frequently occurring log messages as separate groups. All others are plotted as one additional group others.

### Plot Types

Scatter Plot

--type scatter (default)
A scatter plot prints a marker on a two-dimensional axis, where the x-axis represents date and time, and the y-axis represents a certain numeric value. The numeric value for the y-axis can be chosen with an additional parameter (--yaxis, see below). By default, scatter plots show the duration of operations (queries, updates, inserts, deletes, ...) on the y-axis.

Available Groupings

Scatter plots use colors and additionally different marker shapes (circles, squares, diamonds, ...) to display different groups. The supported groupings for scatter plots are: namespace, operation, thread, filename (for multiple files), and regular expressions.

Additional Parameters

Y-Axis Value

--yaxis FIELD

This parameter determines what value should be considered for the location on the y-axis. By default, the y-axis plots duration. Other possibilities are nscanned, nupdated, ninserted, ntoreturn, nreturned, numYields, r (read lock), w (write lock).

Example

mplotqueries mongod.log --type scatter --yaxis w

This command plots the time (x-axis) vs. the write lock values of all operations (y-axis). Only lines that have a write lock value present are considered for the plot. Note that the unit for read/write lock is in microseconds.

#### Scan Ratio Plot

--type nscanned/n
The scan ratio plot is a special type of scatter plot. Instead of plotting a single field as the standard scatter plot, it will calculate the ratio between the nscanned value and the nreturned value, and uses that result as the value for the y-axis. This plot is very useful to quickly find inefficient queries.

Example

mplotqueries mongod.log --type nscanned/n

#### Histogram Plot

--type histogram
Histogram plots don't consider a particular value in the log line (like for example scatter plots do), but rather bin the occurrence of log lines together in time buckets and present the result as a bar chart. The more occurrences of a certain log line (per group) in a given time frame, the higher the bar for that bucket. The size of a bucket is 60 seconds by default, but can be configured to another value (--bucketsize, see below). Unless one wants to know the total number of log lines per time bucket (which is not very useful information), this command should always be preceeded with a filter, for example mlogfilter or grep.

Available Groupings

Histogram plots use colors to display different groups. Each group gets its own bar, the bars are stacked on top of each other to also give an indication of the total number of matched lines per bucket. The supported groupings for histogram plots are: namespace, operation, thread, filename (for multiple files), log2code and regular expressions.

Additional Parameters

Bucket Size

--bucketsize SIZE, -b SIZE (alias)

This parameter sets the bucket size for an individual bucket (bar). The unit is measured in seconds and the default value is 60 seconds. This needs to be adjusted if the total time span of a log file is rather large. More than 1000 buckets are slow to render, and mplotqueries will output a warning to consider increasing the bucket size.

Example

mlogfilter mongod.log --operation insert | 
  mplotqueries --type histogram --bucketsize 3600

This command plots the inserts per hour (3600 seconds) as a histogram plot. By default, the grouping is on namespace.

See this plot for another histogram plot example.

#### Connection Churn Plot

--type connchurn
A connection churn plot is a special plot that only considers lines about opening and closing connections. It will then create an opened (green bars) vs. closed (red bars) plot over time, and additionally show the number of currently open connections (black line, only for MongoDB log files >= 2.2).

Available Groupings

No groupings are supported by this type of plot.

Additional Parameters

Bucket Size

--bucketsize SIZE, -b SIZE (alias)

As with histogram plots, this parameter sets the bucket size for an individual bucket (bar). The unit is measured in seconds and the default value is 60 seconds. This needs to be adjusted if the total time span of a log file is rather large. More than 1000 buckets are slow to render, and mplotqueries will output a warning to consider increasing the bucket size.

Example

mplotqueries --type connchurn --bucketsize 600

This command plots connection churn per 10 minutes (600 seconds) as a bi-directional histogram plot, as well as the total number of open connections at each time (black line).

See this plot for another histogram plot example.

#### Range Plot
#### Event Plot
#### Replica Set State Plot
### Overlays

--overlay add

--overlay list

--overlay reset