monitoring.txt

======================
Monitoring for MongoDB
======================

.. default-domain:: mongodb

.. contents:: On this page
   :local:
   :backlinks: none
   :depth: 1
   :class: singlecol

Monitoring is a critical component of all database administration. A
firm grasp of MongoDB's reporting will allow you to assess the state
of your database and maintain your deployment without crisis.
Additionally, a sense of MongoDB's normal operational parameters will
allow you to diagnose before they escalate to failures.

This document presents an overview of the available monitoring utilities
and the reporting statistics
available in MongoDB. It also introduces diagnostic strategies
and suggestions for monitoring replica sets and
sharded clusters.

.. note::

   |mms-home| is a hosted monitoring service which collects and
   aggregates diagnostic data to provide insight into the performance
   and operation of MongoDB deployments. See |mms-home| and the
   |mms-docs| for more information.

.. include:: /includes/replacement-mms.rst

Monitoring Strategies
---------------------

There are three methods for collecting data about the state of a
running MongoDB instance:

- First, there is a set of utilities distributed with MongoDB that
  provides real-time reporting of database activities.

- Second, :doc:`database commands </reference/command>` return
  statistics regarding the current database state with greater
  fidelity.

- Third, |mms-home| collects data from running MongoDB deployments and
  provides visualization and alerts based on that data.

Each strategy can help answer different questions and is useful in
different contexts. These methods are complementary.

MongoDB Reporting Tools
-----------------------

This section provides an overview of the reporting methods distributed
with MongoDB. It also offers examples of the kinds of questions that
each method is best suited to help you address.

Utilities
~~~~~~~~~

The MongoDB distribution includes a number of utilities that quickly
return statistics about instances' performance and activity. Typically,
these are most useful for diagnosing issues and assessing normal
operation.

``mongostat``
`````````````

:binary:`~bin.mongostat` captures and returns the counts of database
operations by type (e.g. insert, query, update, delete, etc.). These
counts report on the load distribution on the server.

Use :binary:`~bin.mongostat` to understand the distribution of operation types
and to inform capacity planning. See the :doc:`mongostat manual
</reference/program/mongostat>` for details.

``mongotop``
````````````

:binary:`~bin.mongotop` tracks and reports the current read and write
activity of a MongoDB instance, and reports these statistics on a per
collection basis.

Use :binary:`~bin.mongotop` to check if your database activity and use
match your expectations. See the :doc:`mongotop manual
</reference/program/mongotop>` for details.

.. _rest-interface:

REST Interface
``````````````

MongoDB provides a simple REST interface that can be useful for configuring
monitoring and alert scripts, and for other administrative tasks.

To enable, configure :binary:`~bin.mongod` to use :term:`REST`, either by
starting :binary:`~bin.mongod` with the :setting:`--rest <REST>` option,
or by setting the :setting:`rest` setting to ``true`` in a
:doc:`configuration file </reference/configuration-options/>`.

For more information on using the REST Interface see, the
:ecosystem:`Simple REST Interface </tools/http-interfaces>`
documentation.

.. _http-console:

HTTP Console
````````````

MongoDB provides a web interface that exposes diagnostic
and monitoring information in a simple web page. The web interface is
accessible at ``localhost:<port>``, where the
``<port>`` number is **1000** more than the :binary:`~bin.mongod` port .

For example, if a locally running :binary:`~bin.mongod` is using the
default port ``27017``, access the HTTP console at
``http://localhost:28017``.

Commands
~~~~~~~~

MongoDB includes a number of commands that report on the state of the
database.

These data may provide a finer level of granularity than the utilities
discussed above. Consider using their output in scripts and programs to
develop custom alerts, or to modify the behavior of your application in
response to the activity of your instance. The :method:`db.currentOp`
method is another useful tool for identifying the database instance's
in-progress operations.

``serverStatus``
````````````````

The :dbcommand:`serverStatus` command, or :method:`db.serverStatus()`
from the shell, returns a general overview of the status of the
database, detailing disk usage, memory use, connection, journaling,
and index access. The command returns quickly and does not impact
MongoDB performance.

:dbcommand:`serverStatus` outputs an account of the state of a MongoDB
instance. This command is rarely run directly. In most cases, the data
is more meaningful when aggregated, as one would see with monitoring
tools including |mms-home|.
Nevertheless, all administrators should be familiar with the data
provided by :dbcommand:`serverStatus`.

``dbStats``
```````````

The :dbcommand:`dbStats` command, or :method:`db.stats()` from the shell,
returns a document that addresses storage use and data volumes. The
:dbcommand:`dbStats` reflect the amount of
storage used, the quantity of data contained in the database, and
object, collection, and index counters.

Use this data to monitor the state and storage capacity
of a specific database. This output also allows you to compare
use between databases and to determine the average
:term:`document` size in a database.

``collStats``
`````````````

The :dbcommand:`collStats` provides
statistics that resemble :dbcommand:`dbStats` on the collection level,
including a count of the objects in the collection, the size of
the collection, the amount of disk space used by the collection, and
information about its indexes.

``replSetGetStatus``
````````````````````

The :dbcommand:`replSetGetStatus` command (:method:`rs.status()` from
the shell) returns an overview of your replica set's status. The :doc:`replSetGetStatus
</reference/command/replSetGetStatus>` document details the
state and configuration of the replica set and statistics about its members.

Use this data to ensure that replication is properly configured,
and to check the connections between the current host and the other members
of the replica set.

Third Party Tools
~~~~~~~~~~~~~~~~~

A number of third party monitoring tools have support for MongoDB,
either directly, or through their own plugins.

Self Hosted Monitoring Tools
````````````````````````````

These are monitoring tools that you must install, configure and maintain
on your own servers. Most are open source.

.. list-table::
   :header-rows: 1

   * - **Tool**

     - **Plugin**

     - **Description**

   * - `Ganglia <http://sourceforge.net/apps/trac/ganglia/wiki>`_

     - `mongodb-ganglia <https://github.com/quiiver/mongodb-ganglia>`_

     - Python script to report operations per second, memory usage,
       btree statistics, master/slave status and current connections.

   * - Ganglia

     - `gmond_python_modules <https://github.com/ganglia/gmond_python_modules>`_

     - Parses output from the :dbcommand:`serverStatus` and
       :dbcommand:`replSetGetStatus` commands.

   * - `Motop <https://github.com/tart/motop>`_
     - *None*

     - Realtime monitoring tool for MongoDB servers. Shows
       current operations ordered by durations every second.

   * - `mtop <https://github.com/beaufour/mtop>`_

     - *None*

     - A top like tool.

   * - `Munin <http://munin-monitoring.org/>`_

     - `mongo-munin <https://github.com/erh/mongo-munin>`_

     - Retrieves server statistics.

   * - Munin

     - `mongomon <https://github.com/pcdummy/mongomon>`_

     - Retrieves collection statistics (sizes, index sizes, and each
       (configured) collection count for one DB).

   * - Munin

     - `munin-plugins Ubuntu PPA
       <https://launchpad.net/~chris-lea/+archive/munin-plugins>`_

     - Some additional munin plugins not in the main distribution.

   * - `Nagios <http://www.nagios.org/>`_

     - `nagios-plugin-mongodb
       <https://github.com/mzupan/nagios-plugin-mongodb>`_

     - A simple Nagios check script, written in Python.

Also consider `dex <https://github.com/mongolab/dex>`_, an index and
query analyzing tool for MongoDB that compares MongoDB log files and
indexes to make indexing recommendations.

.. seealso:: 
   :products:`Ops Manager, an on-premise solution available in MongoDB
   Enterprise Advanced </mongodb-enterprise-advanced?jmp=docs>`.
   
Hosted (SaaS) Monitoring Tools
``````````````````````````````

These are monitoring tools provided as a hosted service, usually through
a paid subscription.

.. list-table::
   :header-rows: 1

   * - **Name**

     - **Notes**

   * - |mms-home|

     - |MMS| is a cloud-based suite of services for managing MongoDB
       deployments. |MMS| provides monitoring and backup
       functionality.

   * - `Scout <http://scoutapp.com>`_

     - Several plugins, including `MongoDB Monitoring
       <https://scoutapp.com/plugin_urls/391-mongodb-monitoring>`_,
       `MongoDB Slow Queries
       <http://scoutapp.com/plugin_urls/291-mongodb-slow-queries>`_,
       and `MongoDB Replica Set Monitoring
       <http://scoutapp.com/plugin_urls/2251-mongodb-replica-set-monitoring>`_.

   * - `Server Density <http://www.serverdensity.com>`_

     - `Dashboard for MongoDB
       <http://www.serverdensity.com/mongodb-monitoring/>`_, MongoDB
       specific alerts, replication failover timeline and iPhone, iPad
       and Android mobile apps.

.. _stdout:
.. _standard-output:
.. _monitoring-standard-loggging:

Process Logging
---------------

During normal operation, :binary:`~bin.mongod` and :binary:`~bin.mongos`
instances report a live account of all server activity and operations
to either
standard output or a log file. The following runtime settings
control these options.

- :setting:`quiet`. Limits the amount of information written to the
  log or output.

- :setting:`verbose`. Increases the amount of information written to
  the log or output.

  You can also specify this as ``v`` (as in ``-v``). For higher levels of verbosity,
  set multiple ``v``, as in ``vvvv = True``. You can
  also change the verbosity of a running :binary:`~bin.mongod` or
  :binary:`~bin.mongos` instance with the :dbcommand:`setParameter`
  command.

- :setting:`logpath`. Enables logging to a file, rather than the standard
  output. You must specify the full path to the log file when adjusting
  this setting.

- :setting:`logappend`. Adds information to a log
  file instead of overwriting the file.

.. note::

   You can specify these configuration operations as the command line
   arguments to :doc:`mongod </reference/program/mongod>` or :doc:`mongos
   </reference/program/mongos>`

   For example:

   .. code-block:: javascript

      mongod -v --logpath /var/log/mongodb/server1.log --logappend

   Starts a :binary:`~bin.mongod` instance in :setting:`verbose` mode,
   appending data to the log file at
   ``/var/log/mongodb/server1.log/``.

The following :term:`database commands <database command>` also
affect logging:

- :dbcommand:`getLog`. Displays recent messages from the
  :binary:`~bin.mongod` process log.

- :dbcommand:`logRotate`. Rotates the log files for :binary:`~bin.mongod`
  processes only. See :doc:`/tutorial/rotate-log-files`.

Diagnosing Performance Issues
-----------------------------

Degraded performance in MongoDB
is typically a function of the relationship between the
quantity of data stored in the database, the amount of system RAM, the
number of connections to the database, and the amount of time the
database spends in a locked state.

In some cases performance issues may be transient and related to
traffic load, data access patterns, or the availability of hardware on
the host system for virtualized environments. Some users also
experience performance limitations as a result of inadequate or
inappropriate indexing strategies, or as a consequence of poor schema
design patterns. In other situations, performance issues may indicate
that the database may be operating at capacity and that it is time to
add additional capacity to the database.

The following are some causes of degraded performance in MongoDB.

Locks
~~~~~

MongoDB uses a locking system to ensure data set validity. However, if
certain operations are long-running, or a queue forms, performance
will slow as requests and operations wait for the lock. Lock-related
slowdowns can be intermittent. To see if the lock has been affecting
your performance, look to the data in the
:ref:`globalLock` section of the :dbcommand:`serverStatus` output. If
:data:`globalLock.currentQueue.total
<serverStatus.globalLock.currentQueue.total>` is consistently high,
then there is a chance that a large number of requests are waiting for
a lock. This indicates a possible concurrency issue that may be affecting
performance.

If :data:`globalLock.totalTime <serverStatus.globalLock.totalTime>` is
high relative to :data:`~serverStatus.uptime`, the database has
existed in a lock state for a significant amount of time.

Long queries are often the result of a number of factors:
ineffective use of indexes, non-optimal schema design, poor query
structure, system architecture issues, or insufficient RAM resulting
in :ref:`page faults <administration-monitoring-page-faults>` and disk
reads.

Memory Usage
~~~~~~~~~~~~

MongoDB uses memory mapped files to store data. Given a data
set of sufficient size, the MongoDB process will allocate all
available memory on the system for its use.
While this is part of the design, and affords MongoDB superior
performance, the memory mapped files make it difficult to determine if
the amount of RAM is sufficient for the data set.

The :ref:`memory usage statuses <memory-status>` metrics of the
:dbcommand:`serverStatus` output can provide insight into MongoDB's
memory use. Check the resident memory use
(i.e. :data:`mem.resident <serverStatus.mem.resident>`): if this
exceeds the amount of system memory *and* there is a significant amount
of data on disk that isn't in RAM, you may have exceeded the capacity
of your system.

You should also check the amount of mapped memory (i.e. :data:`mem.mapped
<serverStatus.mem.mapped>`.) If this value is greater than the amount
of system memory, some operations will require disk access :term:`page
faults <page fault>` to read data from virtual memory and negatively
affect performance.

.. _administration-monitoring-page-faults:

Page Faults
~~~~~~~~~~~

A page fault occurs when MongoDB requires data
not located in physical memory, and must read from virtual memory. To
check for page faults, see the :data:`extra_info.page_faults
<serverStatus.extra_info.page_faults>` value in the
:dbcommand:`serverStatus` output. This data is only available on
Linux systems.

.. we should reverify the previous statement as the info is appearing
   on OS X as well

A single page fault completes quickly and is not problematic. However, in
aggregate, large volumes of page faults typically indicate that MongoDB
is reading too much data from disk. In many situations, MongoDB's
read locks will "yield" after a page fault to allow other processes to
read and avoid blocking while waiting for the next page to read into
memory. This approach improves concurrency, and also improves overall
throughput in high volume systems.

Increasing the amount of RAM accessible to MongoDB may
help reduce the number of page faults. If this is not possible, you
may want to consider deploying a :term:`sharded cluster` and/or
adding :term:`shards <shard>` to your deployment to
distribute load among :binary:`~bin.mongod` instances.

Number of Connections
~~~~~~~~~~~~~~~~~~~~~

In some cases, the number of connections between the application layer
(i.e. clients) and the database can overwhelm the ability of the
server to handle requests. This can produce performance
irregularities. The following fields in the :doc:`serverStatus
</reference/command/serverStatus>` document can provide insight:

- :data:`globalLock.activeClients
  <serverStatus.globalLock.activeClients>` contains a counter of the
  total number of clients with active operations in progress or
  queued.

- :data:`~serverStatus.connections` is a container for the following
  two fields:

  - :data:`~serverStatus.connections.current` the total number of
    current clients that connect to the database instance.

  - :data:`~serverStatus.connections.available` the total number of
    unused collections available for new clients.

.. note::

   Unless constrained by system-wide limits MongoDB has a hard connection
   limit of 20,000 connections. You can modify system limits
   using the ``ulimit`` command, or by editing your system's
   ``/etc/sysctl`` file.

If requests are high because there are numerous concurrent application
requests, the database may have trouble keeping up with demand. If
this is the case, then you will need to increase the capacity of your
deployment. For read-heavy applications increase the size of your
:term:`replica set` and distribute read operations to
:term:`secondary` members. For write heavy applications, deploy
:term:`sharding` and add one or more :term:`shards <shard>` to a
:term:`sharded cluster` to distribute load among :binary:`~bin.mongod`
instances.

Spikes in the number of connections can also be the result of
application or driver errors. All of the officially supported MongoDB
drivers implement connection pooling, which allows clients to use and
reuse connections more efficiently. Extremely high numbers of
connections, particularly without corresponding workload is often
indicative of a driver or other configuration error.

.. _database-profiling:

Database Profiling
~~~~~~~~~~~~~~~~~~

MongoDB's "Profiler" is a database profiling system that can help identify
inefficient queries and operations.

The following profiling levels are available:

.. list-table::
   :header-rows: 1

   * - **Level**

     - **Setting**

   * - 0

     - Off. No profiling

   * - 1

     - On. Only includes *"slow"* operations

   * - 2

     - On. Includes *all* operations

Enable the profiler by setting the
:dbcommand:`profile` value using the following command in the
:binary:`~bin.mongo` shell:

.. code-block:: javascript

   db.setProfilingLevel(1)

The :setting:`slowms` setting defines what constitutes a "slow"
operation. To set the threshold above which the profiler considers
operations "slow" (and thus, included in the level ``1`` profiling
data), you can configure :setting:`slowms` at runtime as an argument to
the :method:`db.setProfilingLevel()` operation.

.. see:: The documentation of :method:`db.setProfilingLevel()` for more
   information about this command.

By default, :binary:`~bin.mongod` records all "slow" queries to its
:setting:`log <logpath>`, as defined by :setting:`slowms`.

.. note::

   Because the database profiler can negatively impact
   performance, only enable profiling for strategic intervals and as
   minimally as possible on production systems.

   You may enable profiling on a per-:binary:`~bin.mongod` basis. This
   setting will not propagate across a :term:`replica set` or
   :term:`sharded cluster`.

You can view the output of the profiler in the ``system.profile``
collection of your database by issuing the ``show profile`` command in
the :binary:`~bin.mongo` shell, or with the following operation:

.. code-block:: javascript

   db.system.profile.find( { millis : { $gt : 100 } } )

This returns all operations that lasted longer than 100 milliseconds.
Ensure that the value specified here (``100``, in this example) is above the
:setting:`slowms` threshold.

.. seealso:: :doc:`/administration/optimization` addresses strategies
   that may improve the performance of your database queries and
   operations.

.. _replica-set-monitoring:

Replication and Monitoring
--------------------------

Beyond the basic monitoring requirements for any MongoDB instance, for
replica sets, administrators must monitor *replication
lag*. "Replication lag" refers to the amount of time that it takes to
copy (i.e. replicate) a write operation on the :term:`primary` to a
:term:`secondary`. Some small delay period may be acceptable, but two
significant problems emerge as replication lag grows:

- First, operations that occurred during the period of lag are not
  replicated to one or more secondaries. If you're using replication
  to ensure data persistence, exceptionally long delays may impact the
  integrity of your data set.

- Second, if the replication lag exceeds the length of the operation
  log (:term:`oplog`) then MongoDB will have to perform an initial
  sync on the secondary, copying all data from the :term:`primary` and
  rebuilding all indexes. This is uncommon under normal circumstances,
  but if you configure the oplog to be smaller than the default,
  the issue can arise.

  .. note::

     The size of the oplog is only configurable during the first
     run using the :option:`--oplogSize <mongod --oplogSize>` argument to
     the :binary:`~bin.mongod` command, or preferably, the :setting:`oplogSize`
     in the MongoDB configuration file. If you do not specify this on the
     command line before running with the :option:`--replSet <mongod --replSet>`
     option, :binary:`~bin.mongod` will create a default sized oplog.

     By default, the oplog is 5 percent of total available disk space
     on 64-bit systems. For more information about changing the oplog
     size, see the :doc:`/tutorial/change-oplog-size`

For causes of replication lag, see :ref:`Replication Lag
<replica-set-replication-lag>`.

Replication issues are most often the result of network connectivity
issues between members, or the result of a :term:`primary` that does not
have the resources to support application and replication traffic. To
check the status of a replica, use the :dbcommand:`replSetGetStatus` or
the following helper in the shell:

.. code-block:: javascript

   rs.status()

The :doc:`/reference/command/replSetGetStatus` document provides a more in-depth
overview view of this output. In general, watch the value of
:data:`~replSetGetStatus.members.optimeDate`, and pay particular attention
to the time difference between the :term:`primary` and the
:term:`secondary` members.

Sharding and Monitoring
-----------------------

In most cases, the components of :term:`sharded clusters <sharded cluster>`
benefit from the same monitoring and analysis as all other MongoDB
instances. In addition, clusters require further monitoring to ensure
that data is effectively distributed among nodes and that sharding
operations are functioning appropriately.

.. seealso:: See the :doc:`/core/sharding` documentation for more
   information.

Config Servers
~~~~~~~~~~~~~~

The :term:`config database` maintains a map identifying which
documents are on which shards. The cluster updates this map as
:term:`chunks <chunk>` move between shards. When a configuration
server becomes inaccessible, certain sharding operations become
unavailable, such as moving chunks and starting :binary:`~bin.mongos`
instances. However, clusters remain accessible from already-running
:binary:`~bin.mongos` instances.

Because inaccessible configuration servers can seriously impact
the availability of a sharded cluster, you should monitor your
configuration servers to ensure that the cluster remains well
balanced and that :binary:`~bin.mongos` instances can restart.

|mms-home| monitors config servers and can create notifications if a
config server becomes inaccessible. See the |mms-docs| for more
information.

Balancing and Chunk Distribution
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The most effective :term:`sharded cluster` deployments evenly balance
:term:`chunks <chunk>` among the shards. To facilitate this, MongoDB
has a background :term:`balancer` process that distributes data to ensure that
chunks are always optimally distributed among the :term:`shards <shard>`.

Issue the :method:`db.printShardingStatus()` or :method:`sh.status()`
command to the :binary:`~bin.mongos` by way of the :binary:`~bin.mongo`
shell. This returns an overview of the entire cluster including the
database name, and a list of the chunks.

Stale Locks
~~~~~~~~~~~

In nearly every case, all locks used by the balancer are automatically
released when they become stale. However, because any long lasting
lock can block future balancing, it's important to ensure that all
locks are legitimate. To check the lock status of the database,
connect to a :binary:`~bin.mongos` instance using the :binary:`~bin.mongo`
shell. Issue the following command sequence to switch to the
``config`` database and display all outstanding locks on the shard database:

.. code-block:: javascript

   use config
   db.locks.find()

For active deployments, the above query can provide insights.
The balancing process, which originates on a randomly selected
:binary:`~bin.mongos`, takes a special "balancer" lock that prevents other
balancing activity from transpiring. Use the following command, also
to the ``config`` database, to check the status of the "balancer"
lock.

.. code-block:: javascript

   db.locks.find( { _id : "balancer" } )

If this lock exists, make sure that the balancer process is actively
using this lock.