Fetching contributors…
Cannot retrieve contributors at this time
434 lines (313 sloc) 16.8 KB
.. _ec2-backup-and-restore:
EC2 Backup and Restore
.. default-domain:: mongodb
.. contents:: On this page
:backlinks: none
:depth: 1
:class: twocols
This page describes how to backup, verify, and restore a MongoDB running
on EC2 using `EBS Snapshots <>`_.
.. seealso::
`Cloud Manager
orchestrates critical operational tasks you currently perform
manually across the servers in your MongoDB deployment. Download a
free trial of `Cloud Manager
<>`_ to
automatically provision your EC2 instances.
How you backup MongoDB will depend on whether you are using the
``--journal`` option, which is available in versions 1.8 and above.
Backup with ``--journal``
The journal file allows for roll forward recovery. The journal files are
located in the dbpath directory so will be snapshotted at the same time
as the database files.
If the dbpath is mapped to a single EBS volume then proceed to
If the dbpath is mapped to multiple EBS volumes, then in order to guarantee
the stability of the filesystem you will need to :ref:`ec2-flush-and-lock-database`.
.. note:: Snapshotting with the journal is only possible if the journal
resides on the same volume as the data files, so that one snapshot
operation captures the journal state and data file state atomically.
Backup without ``--journal``
In order to correctly backup a MongoDB, you need to ensure that writes
are suspended to the filesystem before you backup the filesystem. If
writes are not suspended then the backup may contain partially written
or data which may fail to restore correctly.
Depending on your version of MongoDB, use
:manual:`fsync and lock </reference/command/fsync/>`
or, after MongoDB 2.0, use
:manual:`db.fsyncLock() </reference/method/db.fsyncLock/>`.
If the filesystem is being used only by the database, then you can
use the snapshot facility of EBS volumes to create a backup. If you are
using the volume for any other application then you will need to ensure
that the filesystem is frozen as well (e.g. on XFS filesystem use
``xfs_freeze``) before you initiate the EBS snapshot.
The overall process looks like:
.. image:: /figures/ec2-backup-and-restore-backup.jpg
:alt: Steps to create a backup using the snapshot facility of EBS volumes
:width: 700px
.. _ec2-flush-and-lock-database:
Flush and Lock the Database
Writes have to be suspended to the filesystem in order to make a stable
copy of the database files.
Prior to MongoDB version 2.0, this is achieved through the MongoDB
shell using
:manual:`fsync and lock </reference/command/fsync/>`:
.. code-block:: sh
mongo shell> use admin
mongo shell> db.runCommand({fsync:1,lock:1});
"info" : "now locked against writes, use db.$cmd.sys.unlock.findOne() to unlock",
"ok" : 1
MongoDB 2.0 added the
:manual:`db.fsyncLock() </reference/method/db.fsyncLock/>`
method to lock the database and flush writes to disk and added the
:manual:`db.fsyncUnlock() </reference/method/db.fsyncUnlock/>`
method to unlock the database after the snapshot has completed.
During the time the database is locked, any write requests that this
database receives will be rejected. Any application code will need to
deal with these errors appropriately.
.. _ec2-backup-database-files:
Backup the Database Files
There are several ways to create an EBS Snapshot. The following examples use the
`AWS command line <>`_
Find the EBS Volumes Associated with MongoDB
If the mapping of EBS Block devices to the MongoDB data volumes is
already known, then this step can be skipped. The example below shows
how to determine the mapping for an LVM volume, please confirm with your
System Administrator how the original system was setup if you are
Find the EBS Block Devices Associated with the Running Instance
.. code-block:: sh
shell> ec2-describe-instances
RESERVATION r-eb09aa81 289727918005 tokyo,default
INSTANCE i-78803e15 ami-4b4ba522 ip-10-204-215-62.ec2.internal running scaleout 0 m1.large 2010-11-04T02:15:34+0000 us-east-1a aki-0b4aa462 monitoring-disabled ebs paravirtual
BLOCKDEVICE /dev/sda1 vol-6ce9f105 2010-11-04T02:15:43.000Z
BLOCKDEVICE /dev/sdf vol-96e8f0ff 2010-11-04T02:15:43.000Z
BLOCKDEVICE /dev/sdh vol-90e8f0f9 2010-11-04T02:15:43.000Z
BLOCKDEVICE /dev/sdg vol-68e9f101 2010-11-04T02:15:43.000Z
BLOCKDEVICE /dev/sdi vol-94e8f0fd 2010-11-04T02:15:43.000Z
As can be seen in this example, there are a number of block devices
associated with this instance. You must determine which volumes make
up the filesystem to snapshot.
Determine how the dbpath is Mapped to the File System
Log onto the running MongoDB instance in EC2. To determine where the
database file are located, either look at the startup parameters for the
:program:`mongod` process or if :program:`mongod` is running, then you
can examine the running process. In the following example, ``dbpath`` is
set to ``/var/lib/mongodb/tokyo0``.
.. code-block:: sh
root> ps -ef | grep mongo
ubuntu 10542 1 0 02:17 ? 00:00:00 /var/opt/mongodb/current/bin/mongod --port 27000 --shardsvr --dbpath /var/lib/mongodb/tokyo0 --fork --logpath /var/opt/mongodb/log/server.log --logappend --rest
Map the dbpath to the Physical Devices
Using the ``df`` command, determine what the ``--dbpath`` directory is mapped to
.. code-block:: sh
root> df /var/lib/mongodb/tokyo0
Filesystem 1K-blocks Used Available Use% Mounted on
104802308 4320 104797988 1% /var/lib/mongodb
Next determine the logical volume associated with this device. In the
example above, this is ``/dev/mapper/data_vg-data_vol``.
.. code-block:: sh
root> lvdisplay /dev/mapper/data_vg-data_vol
--- Logical volume ---
LV Name /dev/data_vg/data_vol
VG Name data_vg
LV UUID fixOyX-6Aiw-PnBA-i2bp-ovUc-u9uu-TGvjxl
LV Write Access read/write
LV Status available
# open 1
LV Size 100.00 GiB
This output indicates the volume group associated with this logical
volume, in this example ``data_vg``. Next determine how this maps to the
physical volume.
.. code-block:: sh
root> pvscan
PV /dev/md0 VG data_vg lvm2 [100.00 GiB / 0 free]
Total: 1 [100.00 GiB] / in use: 1 [100.00 GiB] / in no VG: 0 [0 ]
From the physical volume, determine the associated physical devices, in
this example ``/dev/md0``.
.. code-block:: sh
root> mdadm --detail /dev/md0
Version : 00.90
Creation Time : Thu Nov 4 02:17:11 2010
Raid Level : raid10
Array Size : 104857472 (100.00 GiB 107.37 GB)
Used Dev Size : 52428736 (50.00 GiB 53.69 GB)
Raid Devices : 4
UUID : 07552c4d:6c11c875:e5a1de64:a9c2f2fc (local to host ip-10-204-215-62)
Events : 0.19
Number Major Minor RaidDevice State
0 8 80 0 active sync /dev/sdf
1 8 96 1 active sync /dev/sdg
2 8 112 2 active sync /dev/sdh
3 8 128 3 active sync /dev/sdi
The block devices ``/dev/sdf`` through ``/dev/sdi`` make up this
physical devices. Each of these volumes will need to be snapped in order
to complete the backup of the filesystem.
.. _create-ebs-snapshot:
Create the EBS Snapshot
Create the snapshot for each device. Using the
`ec2-create-snapshot <>`_
command, use the Volume Id for the device listed by the
`ec2-describe-instances <>`_
.. code-block:: sh
shell> ec2-create-snapshot -d backup-20101103 vol-96e8f0ff
SNAPSHOT snap-417af82b vol-96e8f0ff pending 2010-11-04T05:57:29+0000 289727918005 50 backup-20101103
shell> ec2-create-snapshot -d backup-20101103 vol-90e8f0f9
SNAPSHOT snap-5b7af831 vol-90e8f0f9 pending 2010-11-04T05:57:35+0000 289727918005 50 backup-20101103
shell> ec2-create-snapshot -d backup-20101103 vol-68e9f101
SNAPSHOT snap-577af83d vol-68e9f101 pending 2010-11-04T05:57:42+0000 289727918005 50 backup-20101103
shell> ec2-create-snapshot -d backup-20101103 vol-94e8f0fd
SNAPSHOT snap-2d7af847 vol-94e8f0fd pending 2010-11-04T05:57:49+0000 289727918005 50 backup-20101103
Unlock the Database
Once the snapshots are in ``pending`` state, the
database can be unlocked. Use the following operation to unlock the database.
.. code-block:: javascript
mongo shell> db.$cmd.sys.unlock.findOne();
{ "ok" : 1, "info" : "unlock requested" }
When unlocked, the database is available to process write requests.
Verify the Backup
In order to verify the backup, you must do the following:
- Check the status of each snapshot to ensure it is "completed."
- Create new volumes based on the snapshots and mount the new volumes.
- Run :program:`mongod` and verify the collections.
.. image:: /figures/ec2-backup-and-restore-verify.jpg
:width: 700px
:alt: Steps to verify the backup
Typically, the verification is performed on another machine so that
you do not burden your production systems with the additional CPU and
I/O load of the verification processing.
Describe the Snapshots
Using the `ec2-describe-snapshots <>`_
command, find the snapshots that make up the backup. Using a filter on
the ``description`` field, snapshots associated with the given backup are
easily found. The search text used should match the text used in the ``-d``
flag passed to `ec2-create-snapshot <>`_ command when the backup was made.
.. code-block:: sh
backup shell> ec2-describe-snapshots --filter "description=backup-20101103"
SNAPSHOT snap-2d7af847 vol-94e8f0fd completed 2010-11-04T05:57:49+0000 100% 289727918005 50 backup-20101103
SNAPSHOT snap-417af82b vol-96e8f0ff completed 2010-11-04T05:57:29+0000 100% 289727918005 50 backup-20101103
SNAPSHOT snap-577af83d vol-68e9f101 completed 2010-11-04T05:57:42+0000 100% 289727918005 50 backup-20101103
SNAPSHOT snap-5b7af831 vol-90e8f0f9 completed 2010-11-04T05:57:35+0000 100% 289727918005 50 backup-20101103
Create New Volumes Based on the Snapshots
Using the `ec2-create-volume <>`_
command, create a new volumes based on each of the snapshots that make
up the backup.
.. code-block:: sh
backup shell> ec2-create-volume --availability-zone us-east-1a --snapshot snap-2d7af847
VOLUME vol-06aab26f 50 snap-2d7af847 us-east-1a creating 2010-11-04T06:44:27+0000
backup shell> ec2-create-volume --availability-zone us-east-1a --snapshot snap-417af82b
VOLUME vol-1caab275 50 snap-417af82b us-east-1a creating 2010-11-04T06:44:38+0000
backup shell> ec2-create-volume --availability-zone us-east-1a --snapshot snap-577af83d
VOLUME vol-12aab27b 50 snap-577af83d us-east-1a creating 2010-11-04T06:44:52+0000
backup shell> ec2-create-volume --availability-zone us-east-1a --snapshot snap-5b7af831
VOLUME vol-caaab2a3 50 snap-5b7af831 us-east-1a creating 2010-11-04T06:45:18+0000
Attach the New Volumes to the Instance
Using the `ec2-attach-volume <>`_
command, attach each volume to the instance where the backup will be
.. code-block:: sh
backup shell> ec2-attach-volume --instance i-cad26ba7 --device /dev/sdp vol-06aab26f
ATTACHMENT vol-06aab26f i-cad26ba7 /dev/sdp attaching 2010-11-04T06:49:32+0000
backup shell> ec2-attach-volume --instance i-cad26ba7 --device /dev/sdq vol-1caab275
ATTACHMENT vol-1caab275 i-cad26ba7 /dev/sdq attaching 2010-11-04T06:49:58+0000
backup shell> ec2-attach-volume --instance i-cad26ba7 --device /dev/sdr vol-12aab27b
ATTACHMENT vol-12aab27b i-cad26ba7 /dev/sdr attaching 2010-11-04T06:50:13+0000
backup shell> ec2-attach-volume --instance i-cad26ba7 --device /dev/sds vol-caaab2a3
ATTACHMENT vol-caaab2a3 i-cad26ba7 /dev/sds attaching 2010-11-04T06:50:25+0000
Mount the Volumes
First make the filesystem visible to the host operating system. This
process will vary based on the particular volume management scheme
that you are using. Check with your system administrator if you are
If you are using Logical Volume Manager, assemble the logical
device from the physical devices. The UUID for the device will be the
same as the original UUID from which the backup was made, and can be
obtained using the ``mdadm`` command.
.. code-block:: sh
backup shell> mdadm --assemble --auto-update-homehost -u
07552c4d:6c11c875:e5a1de64:a9c2f2fc --no-degraded /dev/md0
mdadm: /dev/md0 has been started with 4 drives.
You can confirm that the physical volumes and volume groups appear
correctly to the O/S by executing the following:
.. code-block:: sh
backup shell> pvscan
PV /dev/md0 VG data_vg lvm2 [100.00 GiB / 0 free]
Total: 1 [100.00 GiB] / in use: 1 [100.00 GiB] / in no VG: 0 [0 ]
backup shell> vgscan
Reading all physical volumes. This may take a while...
Found volume group "data_vg" using metadata type lvm2
Next, create the mount point and mount the filesystem:
.. code-block:: sh
backup shell> mkdir -p /var/lib/mongodb
backup shell> cat >> /etc/fstab << EOF
/dev/mapper/data_vg-data_vol /var/lib/mongodb xfs noatime,noexec,nodiratime 0 0
backup shell> mount /var/lib/mongodb
Start the Database
After the filesystem has been mounted, MongoDB can be started. Ensure
that the owner of the files is set to the correct user and group. Since
the backup was made with the database running, the lock file will need
to be removed in order to start the database.
.. code-block:: sh
backup shell> chown -R mongodb /var/lib/mongodb/toyko0
backup shell> rm /var/lib/mongodb/tokyo0/mongod.lock
backup shell> mongod --dbpath /var/lib/mongodb/tokyo0
Verify the Collections
:manual:`Validate </reference/command/validate/>`
each collection to ensure it does not contain any invalid BSON objects.
.. code-block:: javascript
mongo shell> db.blogs.validate({full:true})
Restore uses the same basic steps as the verification process:
.. NOTE The following terse steps are taken from the screenshot (see note after steps).
1. :manual:`db.shutdownServer() </reference/method/db.shutdownServer/>`
2. `ec2-create-volume <>`_
`ec2-attach-volume <>`_.
3. Mount the file system.
4. Run :program:`mongod`.
.. image:: /figures/ec2-backup-and-restore-restore.jpg
:width: 600px
:alt: Basic steps to restore
After the filesystem is mounted you can decide to:
- Copy the database files from the backup into the current database
- Start :program:`mongod` from the new mount point, specifying the new
mount point in the ``--dbpath`` argument.
After the database is started, it will be ready to transact. It will be
at the specific point in time from the backup, so if it is part of a
master/slave or replica set relationship, then the instance will need to
synchronize itself to get itself back up to date.
See Also
|mms-home| for seamless automation, backup, and monitoring.
.. include:: /includes/replacement-mms.rst