[Dogstream] Aggregation of metrics returned by Custom Log Parser

[saganshul]

This is an informative issue or question for dogstreamer.
After hunting the issue for 48hr, I finally went through source code and found all metrics that are parsed by my log parser were not sent to datadog HQ.

The reason for the same is function _aggregate.

Use case where this implementation fails:

Suppose I am parsing Nginx access logs.

My log format is:

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" $request_time $host $upstream_addr $upstream_response_time $pipe;

After parsing the logs I am sending metric:

('nginx.net.status', 1498501864.0, 2.0, {'metric_type': 'counter', 'tags': ['code:500', 'domain:xx.yy.zz', 'upstream:xx.xx.xx.xx:yy']})

Now suppose in DataDog I want to monitor number of status_code:500 per $host. Basically I want to monitor for which $host I am getting status_code:500. This will not be possible since Dogstream is submitting medians based on timestamp and metric_name (deleting most of the data)

This implementation would be better if the function __aggregate also considers tags associated with metric but that would also fail in above usecase.

Now my question is:

1. Why metrics aggregation is necessary, why can't log parser send all the metrics returned?
2. How should I implement my usecase with present implementation?