Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sending metrics to StatsD don't specify any metric type #737

Closed
Hamm-m opened this issue Mar 22, 2018 · 7 comments
Closed

Sending metrics to StatsD don't specify any metric type #737

Hamm-m opened this issue Mar 22, 2018 · 7 comments

Comments

@Hamm-m
Copy link

Hamm-m commented Mar 22, 2018

Since Java Melody 1.70.0, there should be support for sending metrics from Java Melody to StatsD. If I'm not mistaken, it was implemented in issue #678.

The problem I face is that although metrics are send to the StatsD deamon, there are no metric types specified for any of it's datagrams.

There are two negative consequences:

  1. Each datagram send by Java Melody causes "Bad line: 1.00 in msg "javamelody.xxx.hostname.cpu:1.00" error thrown by StatsD as it can't be parsed properly by StatsD deamon.
  2. StatsD deamon don't seem to be able to process datagrams with no metric type specified. These datagrams are counted in "statsd.metrics_received" (and because of (1) also in "statsd.bad_lines_seen") but all the information about their actual value is lost.

The second issue is obviously more severe as it prevents client from using statistic for anything useful. E.g. if you want to track activeThreads, you don't have the actual value of the active threads in hands.

When I experimentally tried to patch Java Melody and changed Statsd.java : line 99 to include e.g. counter metric type at the end of each datagram:
bufferWriter.append(decimalFormat.format(value)).append("|c").append('\n');

..each of the datagram I handled separately as counter and thus I at least see the information about what kind of statistics Java Melody sends. (Obviously not every one of them should be counter metric type)

I believe that each of the datagram send by Java Melody should either have it's metric type pre-specified based on it's logical use in statistics or it should be explicitly configurable by the client. (Or maybe the combination of the two would be ideal, pre-specified types that could be possibly overridden by client)

@evernat
Copy link
Member

evernat commented Mar 25, 2018

It seems that I have forgotten to include the metric type indeed. Sorry.
The metrics should be sent with the gauge metric type for all metrics.
So the line should be:
bufferWriter.append(decimalFormat.format(value)).append("|g").append('\n');
Does it work like that?

@Hamm-m
Copy link
Author

Hamm-m commented Mar 28, 2018

No worries mate!

I'm not entirely sure about the gauge data type to be honest. I'm pretty new in StatsD but my understanding is that the only metric type that automatically does some data-processing is Timer. Counter and Gauge feels to just pass the value to the server and the problem I see with Gauge is that it ignores all except the last value that comes within an interval. So if activeThreads are e.g. 100+ most of the time but then it jumps down to 20 at the end of the interval, the information about high peaks might disappear occasionally.

This article is one of a very few where I read a decent description of the resulted behaviour when using various metric types and after reading it, I'm under impression that Timer might be more useful from client perspective as it includes additional information as avg/min/max etc.

What do you think?

@evernat
Copy link
Member

evernat commented Mar 28, 2018

I still think that the gauge data type should be used, because metrics are sent by javamelody once per minute and can be seen as already aggregated by javamelody. For example, the %cpu metric is averaged for a minute.
So having value and mean and sum and count and lower and upper and 90th percentiles every 10s in StatsD does not make sense. Of course, it depends on the flush interval of StatsD (default 10s) and if this flush interval is greater than the javamelody resolution (default 60s), but I don't think that users would set a flush interval greater than 60s.
So my undertstanding is that Timer is better than Gauge here only if flush interval is greater than 60s. Otherwise Gauge is the same, but with 1 value to store instead of 9.

Is it ok for you?

@Hamm-m
Copy link
Author

Hamm-m commented Mar 28, 2018

Thanks, that makes sense to me. It feels slightly inconsistent with the "recommended" semantic of StatsD metric types but as long as Java Melody makes the aggregation for us, it sounds like we will have transparent results from the client perspective in any case.

In short, sure, go ahead with gauge! :] ..and thanks for quick replies!

@evernat
Copy link
Member

evernat commented Mar 30, 2018

Fixed by 7f4297f and ready for the next release (1.72).
A snapshot build is available at:
https://javamelody.ci.cloudbees.com/job/javamelody/net.bull.javamelody$javamelody-core/

@Hamm-m
Copy link
Author

Hamm-m commented Apr 4, 2018

Thanks! Is there any estimation when 1.72 will be available?

@evernat
Copy link
Member

evernat commented Apr 15, 2018

1.72 is available now

goldyliang pushed a commit to goldyliang/javamelody that referenced this issue Mar 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants