gammaray is a system implementing disk-based introspection for virtual machines. It thus far works with VMs that have disks virtualized via QEMU; however, this limitation is only due to the scarcity of developer time. Conceptually, and practically, gammaray can perform introspection with any source of raw disk writes. The instructions below assume an Ubuntu 12.04 LTS host, although they should be similar for most distributions of Linux.
Follow the non-optional dependencies instructions
Follow the installation procedure for gammaray
Follow the example virtul disk creation instructions
Follow the instructions for running the gammaray pipeline
Check that things are working by using
tailfrom outside the guest on a known text file. Add lines within the guest and ensure that they eventually become visible via the
gray-fsmounted file system.
gray-fs /mnt/disk -d -s disk.raw
/mnt/diskwith whatever folder you'd like to use as the read-only view into the real-time guest file system.
gray-fsdoesn't properly parse command line options yet, so type the command almost exactly as it appears above (
-s -dforce single-threaded, debug mode).
Example check command outside the VM guest
Example check command within the VM guest
while 1; do echo 'test' >> test.txt; sleep 3; done
All source code, documentation, and related artifacts associated with the gammaray open source project are licensed under the Apache License, Version 2.0.
The following libraries are needed to build and execute gammaray (most developers can skip the Python libraries):
hiredis [BSD 3-clause] - the Redis client-side library
sudo apt-get install libhiredis-dev libhiredis0.10
and install a Redis server if you don't have one already:
sudo apt-get install redis-server
event [BSD 3-clause] - the libevent event-driven networking library
sudo apt-get install libevent-dev libevent-2.0.5
bson [Optional, BSD 3-clause] - Python BSON library
redis-py [Optional, MIT] - Python hiredis wrapper
libfuse [GNU LGPL] - the filesystem in userspace library
sudo apt-get install libfuse-dev
The Python libraries are optional if you want to write or execute Python monitors consuming gammaray's publish-subscribe stream of file-level updates.
In addition to
libevent, gammaray requires a slightly
modified version of QEMU. Clone the QEMU repository, apply a patch for
gammaray support, and then compile a gammaray-friendly QEMU.
Get the official QEMU source tree
git clone git://git.qemu-project.org/qemu.git
Checkout a specific tagged commit to apply our patch cleanly
git checkout v1.4.0
Apply the patch (located in
src/patches) to your checked out QEMU tree. This patch is in the gammaray source tree binary_tracing_block_qemu_1-4-0.patch
git apply binary_tracing_block_qemu_1-4-0.patch
Configure QEMU, remember to change the prefix as needed
./configure \ --enable-system \ --disable-user \ --enable-kvm \ --enable-trace-backend=stderr \ --target-list='i386-softmmu i386-linux-user x86_64-linux-user x86_64-softmmu' \ --prefix=/home/wolf/qemu_bin
Make, and make install QEMU.
make -j2 make install
QEMU binaries with the patch compiled in should be within the
prefixfolder, specifically inside the
Ensure all dependencies are installed already
git clone gammaray's source tree
git clone https://github.com/cmusatyalab/gammaray.git
Bootstrap your source tree
If you want to create a statically linked version of everything, use the following
makeoptions (reference: Re: Building all static):
./configure LDFLAGS="-static" LIBS='-lrt' make LDFLAGS=-all-static
If you want to compile with a custom library, and statically link it in for development purposes, the following should work (replace paths):
./configure LDFLAGS='-L/home/wolf/Dropbox/Projects/hiredis -static' LIBS="-lrt" CPPFLAGS="-I/home/wolf/Dropbox/Projects/hiredis" make LDFLAGS="-all-static -L/home/wolf/Dropbox/Projects/hiredis/"
make -j 2
All binaries will now be built and placed in the bin folder at the top-level directory of the project.
[Optional] Run make install (if you want)
sudo make install
Description of Components
gammaray is organized as a set of tools, described below:
gray-crawler- used to index a disk in preparation for introspection
- This tool is usually used offline, although it can be used online as well
gray-ndb-queuer- used to asynchronously queue writes for analysis
- This tool interfaces with a stream of writes, soon-to-be an NBD endpoint
gray-inferencer- used to analyze queued writes and perform introspection
- This tool loads and maintains metadata in-sync with a live disk
gray-fs- used to produce a FUSE file system view of the in-sync metadata
- This tool provides a read-only FS without mounting the real guest FS
The exact steps are enumerated below, but at a high level you must crawl the disk you wish to introspect, load metadata from that crawl for run-time introspection, and attach a copy of the write stream to that disk at run-time to the inferencing backend. Currently, this is coordinated via a named pipe and Redis. In the future, the named pipe is being replaced by NBD. None of the tools auto-daemonize as of this writing.
Crawl the disk that you wish to introspect using
gray-crawler disk.raw disk.bson
Setup a named pipe to receive raw disk writes to the
gray-ndb-queuerand let it read from the named pipe
gray-ndb-queuer disk.bson disk.fifo 4 1>queuer.log 2>queuer.error.log &
gray-inferencerand wait for it to load metadata from
gray-inferencer disk.bson 4 disk_test_instance &
Run QEMU with this disk redirecting stderr output to the named pipe
/path/to/custom/qemu-system-x86_64 \ -enable-kvm \ -cpu kvm64 \ -smp cores=1,threads=1,sockets=1 \ -drive file=disk.raw,if=virtio,aio=native \ -m 1024 \ -display vnc=127.0.0.1:1 \ -trace events=events \ -redir tcp:2222::22 2> disk.fifo &
Remember to use the specially built QEMU when installing dependencies (ie replace the path in the above command). Also, ensure you have an
eventsfile that turns on tracing for the event of interest:
eventsfile's contents should have only that event name on a single line by itself (tracing multiple events will mangle the binary tracing needed for disk introspection).
Example Creation of gammaray-Supported Disk Layout
If you're sufficiently confident with the installer of your OS of choice, feel free to skip the steps below. Otherwise, it might be easier to setup a known good partition configuration with a host system and then install the guest OS into the pre-existing partition.
Use dd or another suitable command to preallocate the raw disk image
dd of=disk.raw seek=$((1024*1024*1024*5)) count=0 bs=1
Create a partition table
parted -s disk.raw mklabel msdos
Create a single primary partition taking up the entire image
parted -s disk.raw mkpart primary ext4 1 $((1024*5))
Make the partition visible to your host as a block device
sudo kpartx -av disk.raw
Format the partition as ext4 [replace loop0 with output from kpartx command]
sudo mkfs.ext4 /dev/mapper/loop0p1
Remove the visible partition and block device from your host
sudo kpartx -dv disk.raw
Boot the instance with install media (example: ubuntu-12.04.2-server-amd64.iso) and the new drive attached
qemu-system-x86_64 disk.raw -cpu kvm64 -cdrom ubuntu-12.04.2-server-amd64.iso
8. Using `Manual Partitioning` at the disk setup stage, select the first partition to be used as `ext4` and `mount point '/'` 9. Then just finish partitioning and continue with the installation procedure ## Currently Supported 1. File Systems * ext4 * NTFS 2. VMMs * KVM/QEMU 3. Number of Disks * Single attached disk 4. Number of File Systems * Single introspected file system 5. Disk Formats * raw