Can't count $bytes_sent or $request_length

[groe]

I want to count incoming/outgoing traffic through statsd. This works:

statsd_count "mysite.outgoing_body_traffic" $body_bytes_sent;

However, $body_bytes_sent does not include HTTP headers. When I try one of the following, nginx won't start:

statsd_count "mysite.outgoing_traffic" $bytes_sent;
statsd_count "mysite.incoming_traffic" $request_length;

The error messages I get:

Restarting nginx: nginx: [emerg] unknown "bytes_sent" variable
Restarting nginx: nginx: [emerg] unknown "request_length" variable

Any ideas why $body_bytes_sent works perfectly but $bytes_sent and $request_length are unknown variables? Logging them to a file (via log_format) works.

Thanks!

[zacman85]

The ngx_log module actually calculates its own set of variables for bytes_sent and request_length. It then merges them in with the standard nginx variables and headers before compiling them into the log format. I have not figured out how to directly access these special ngx_log variables from outside of the ngx_log module.The ngx_statsd module does run in the log phase, so it should have access to all of the same information. It might just be a matter of copying the ngx_log module's variable code into the ngx_statsd module and having ngx_statsd calculate its own versions of those variables.

[groe]

Oh okay, I did not know that there are actual differences in where these variables are coming from (I guess $body_bytes_sent is calculated by nginx core then?).

Unfortunately I don't know a lot about the nginx internals and the module architecture. However, I just had a quick look in the sources and I think these are the functions we are looking for:

// src/http/modules/ngx_http_log_module.c
static u_char *
ngx_http_log_bytes_sent(ngx_http_request_t *r, u_char *buf,
    ngx_http_log_op_t *op)
{
    return ngx_sprintf(buf, "%O", r->connection->sent);
}
static u_char *
ngx_http_log_request_length(ngx_http_request_t *r, u_char *buf,
    ngx_http_log_op_t *op)
{
    return ngx_sprintf(buf, "%O", r->request_length);
}

Would it be much effort to include something like that in nginx-statsd? That would be really great.

Let me know if I can help you (maybe you could give me a starting point or something?).

[zacman85]

It will probably be a week before I will have a chance to work on this. Feel free to send me any code reviews if you want to try your hand at it.

[groe]

Thanks for your answer! I have just investigated both yours and the nginx core code a little and tried to bring ngx_log and nginx-statsd together, so all variables from ngx_log could be used (not only bytes_sent and request_length) in nginx-statsd. Unless we copy over all functions from ngx_log, nginx-statsd would require nginx to be compiled with the ngx_log module enabled. I guess that would be the best option.

I've also had a look on nginx-udplog since they seem to provide support for all ngx_log variables, not only the core ones, and it seems like they depend on the ngx_log variable-compile functions as well:

extern ngx_module_t  ngx_http_log_module;

However, they just have to grab predefined log formats from the ngx_log configuration, they do not define new ones. So I could not find any code for variable compilation there.

As far as I unterstood all that, we currently rely on ngx_http_complex_value for variable interpolation, right? One option could be to simple check whether ngx_http_log_vars (a static array containing all ngx_log variables) contains a value before passing it to ngx_http_complex_value. Unfortunately that would require a few function parameter changes as well and I am not very experienced in C (I just tried to implement that and ended up with three segfaults until I gave up, meh).

If you have a free minute, any help would really be appreciated.

[groe]

Any news on this?

[groe]

Turns out, this is actually something the nginx developers want to fix in future relases (so all variables will be available via complex values). I've submitted a patch that adds $bytes_sent to ngx_http_variables there and it was merged in today. So future nginx releases (>= 1.3.8) should support $bytes_sent (however, there may still be other variables missing, until they are all supported in the normal variable interpolation functions).

Closing here.

[zacman85]

Awesome. Thanks for looking into this. Really sorry I haven't had time to fix. Been a crazy couple of weeks.

On Wednesday, October 3, 2012 at 11:49 AM, groe wrote:

Turns out, this is actually something the nginx developers want to fix in future relases (so all variables will be available via complex values). I've submitted a patch that adds $bytes_sent to ngx_http_variables (http://trac.nginx.org/nginx/changeset/4887/nginx) there and it was merged in today. So future nginx releases (>= 1.3.8) should support $bytes_sent (however, there may still be other variables missing, until they are all supported in the normal variable interpolation functions).
Closing here.

—
Reply to this email directly or view it on GitHub (#1 (comment)).