Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Add DTrace support for Redis #93

Open
videlalvaro opened this Issue Sep 20, 2011 · 5 comments

Comments

Projects
None yet
5 participants

I've been working on adding DTrace support for Redis.

DTrace offers dynamic tracing, so we can debug programs without the need to modify its source code. A design goal of the tool was to be able to debug production systems without affecting their performance. DTrace has been used in production at Sun since 2005 and it's considered "production safe".

Using Dynamic tracing one can be able to trace system calls, see which process is accessing which files, network operations made by programs and more.

More information about it here: http://dtrace.org/blogs/brendan/ and here: http://en.wikipedia.org/wiki/DTrace.

DTrace is supported in Solaris, Mac OS X, FreeBSD and NetBSD. There's a project trying to integrate it to Linux.

Besides from dynamic tracing DTrace offers static tracing. With static tracing we can add our own "probe providers" to our software and hook from there to perform our own tracing, statistics collection and more. MySQL for example fires dtrace probes when is parsing a query and some others: http://dev.mysql.com/tech-resources/articles/getting_started_dtrace_saha.html.

I think it will be nice to have a similar integration into Redis. So for example we could do analysis like this one: Redis Commands execution speed.

There I placed probes that are fired whenever a command is executed as you can see here: videlalvaro/redis@73acc34:

The Dtrace script to produce such output is really simple:

#!/usr/sbin/dtrace -CZs
/*
 * Sample script to calculate the avg time spent inside commands
 * Run it like this: sudo dtrace -qs commands.d -p `pgrep redis`
 * Time output is in nanoseconds.
 */
redis$target:::command-entry
{
  self->ts = timestamp;
}

redis$target:::command-return
/self->ts/
{
  @time[copyinstr(arg0)] = avg(timestamp - self->ts);
  @timeq[copyinstr(arg0)] = quantize(timestamp - self->ts);
  self->ts = 0;
}

Defining probes points is very simple. For example to fire a probe when a command is called I added this in a file called reids_dtrace.d:

provider redis {
    probe command__entry(char *cmd);

    probe command__return(char *cmd);
};

We can have many more probes, is just a matter of specifying there with that simple syntax, for example we could add:

probe client__connected(host, port);

Or simply client__connected with no arguments.

If we run the command:

dtrace -h -s redis_dtrace.d -o redis_dtrace.h

DTrace will generate a header file that defines a couple of macros that is what we use in the code like this:

 int processCommand(redisClient *c) {
    ....

    if(REDIS_COMMAND_ENTRY_ENABLED()) {
       REDIS_COMMAND_ENTRY(c->cmd->name);
    }
    call(c);
    if(REDIS_COMMAND_RETURN_ENABLED()) {
        REDIS_COMMAND_RETURN(c->cmd->name);
    }
    ...
  }

So basically to add probes to Redis we need to define them in redis_dtrace.d and then use the macros wherever we need to fire such probes.

To add this to Redis we need to define useful probes and then add them to the source code at the proper places.

I think having DTrace support can help a lot of people running Redis in production. If we define what probes we want to have I think the job of adding them to Redis shouldn't be more than one work day. Please take a look at my branch to see what I have done so far: https://github.com/videlalvaro/redis/tree/dtrace.

ajuckel commented Sep 21, 2011

(Mostly nothing comment, but github doesn't have a generic "watch this issue" that I can see). DTrace support would be a great addition as having dtrace support throughout the levels of your application allows you to really pinpoint performance issues.

Seldaek commented Sep 21, 2011

+1 on this, more debugging/profiling capabilities are always welcome.

@ajuckel: BTW, below the comment form there is a "Enable notifications for this issue" link, which does just that :)

ajuckel commented Sep 21, 2011

@Seldaek: Ah ha! See it now. Thanks for the tip.

mranney commented Jan 27, 2012

I wish that I could plus more than one for this. Thanks for working on it.

xen0l commented Jun 5, 2016

+1 on this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment