different rrd types

[sbnedkov]

Hello and thanks for creating Metriks.

I'm building a simple CouchDB monitor based on it and wanted to ask whether you plan to add support for different RRD types in the Plugin object - COUNTER vs GAUGE, ability to add a custom step, AREA graphs, etc.

I'll be glad to help in case you don't have the time but have an idea about it happening.

[kvz]

Hi there.

Thanks! The project is still very young, it's my pet project and I only get to work on it on from time to time. I would like to support those though. I've added them to the Todo section in the readme! You mentioned CouchDB monitoring, do you have a specific example / plan that we could aim development towards? I'd love to have real life use-cases and make this a team effort.

Best, Kevin

[sbnedkov]

Hello, thanks for answering.

My goal is to show some CouchDB stats in a convenient way (on a web page) and be able to easily switch them on/off so that I can monitor some issues with an application I'm running. I don't need more than 30 minutes of stats for the moment. CouchDB provides all stats with a simple HTTP request and all that is missing is some visualization.

I've used Metriks this far, tapping into the Plugin object and running updates/creating graphs explicitly and continuously. The other part of my application (couchdb-stats-monitor) is a Node.js server that provides an HTTP API to transfer the graph files. As I needed to have a shared configuration between the two, telling each what exactly is monitored, I wasn't able to integrate fully with Metriks as a plugin.

Would you accept a contribution to develop part of the Node.js API that you mention in the readme that will allow me to do the simple graphing I do now and customize it further? I do now realize that my current usage is not what you had in mind, but hopefully we can agree on what is needed.

[kvz]

Hi, that sounds awesome, I think both your usecase and what I had in mind could be supported via the same API - I'm sure we could reach consensus. Actually if the API is anything like the example in the readme, has some test coverage, and replaces the current way of adding series, I'd be happy to merge a PR.

What do you think?

[sbnedkov]

I think it sounds great. :) If you have more API ideas I'll be glad to hear them or otherwise I'll try to come up with something myself and get back to you. I'm open to either way.

[kvz]

I guess one other thing I had in mind is that it should be extremely easy to add more graphs. Preferably with 1 line of code. So from the end-user/app perspective I think it would be nice if it were as simple as e.g.:

metriks.log('load', 1.6);

or:

metriks.log('instances',  {db: 1, web: 2});

and that would create an instances.rrd with sane defaults if it doesn't exist yet, and then save the 2 datasources in it.

If I would need anything more advanced I could pass options:

metriks.log('instances', {db: 1, web: 2}, {graph: {title: "Running instance types"}});

I guess this would require we save an instances.json configuration file, so that a subsequent:

metriks.log('instances', {db: 1, web: 3});

..would still know about the graph's title.

I'm welcoming any thoughts on this!

[kvz]

PS, I just added a very basic webserver for browsing graphs:

2c66e2a

[sbnedkov]

Hello, sorry for the delay.

This all looks good, I only have some worries about saving the configuration file. There are two sets of configuration that I see - one is related to the RRD and the .rrd file itself is the configuration. The other one is the graph and is volatile, if we want to make it permanent then we indeed need to save its parameters somewhere. Having graph level configuration also means that we need to _.extend(conf, params) each time that the user calls metriks with (graph) options, I guess, so that the user can change parameters and have control over the graph. This means that we need a way to delete parameters not only replace them. Does this make sense?

I would say to have an optional error handler, something like:

metriks.log('load', 1.6);
metriks.log('load', 1.7, {rrd: {type: metriks.counter}}).onError(function (err) {
    console.log(err); // err === "RRD already intialized"
});

I also suggest that the default for rrd, png and configuration files to be the user's home directory, maybe under .metriks/? The remaining defaults will be as they are now for plugins.

What do you say?

[kvz]

Hey,

Your concerns make sense yes, things would get ambiguous, let's not go the magic extend route. Your suggestion looks good though! As for defaults, png & rrd already point to ~/metriks/{png,rrd}. It's just the plugindir that points to ./plugins. I guess what I like about that is that a new user can run metriks and see it work right away, without having to setup a plugin dir and copy files to it. Then when it's time to customise, providing a --plugin-dir seems trivial. I want to optimize for a smooth ride and have things work right out of the box, as I think that's what could set this project apart from the big/advanced ones out there. But maybe I'm overlooking important things.

What do you think?

[kvz]

off-topic: I just registered http://metriks.io/ and put up a very basic site via the gh-pages branch. Just a port of the readme, but we can improve as we go

[kvz]

I had some time and did some work on supporting AREA, and more fine-grained control over datasources in general. Here's it demonstrated in the nslookup.sh plugin:

Because you can now reference dsnames indivdually, I have deprecated . in favor of -> as recursion delimiter as . is commonly used in e.g. ips, and people are likely to use them in dsnames.

let me know what you think

[sbnedkov]

Hi, I'm not an RRD expert so I can't tell for sure but I think you're getting it right and it does work out of the box.

About the new plugin options - they look great. If we add some custom step (--step parameter), a custom number of samples in the RRA (RRA:AVERAGE:0.5:1:17280) and the COUNTER data type then I can have my live monitoring of CouchDB by writing a plugin and have only the front end custom.

I'm still reluctant to do this for one reason, but it's a bit long to explain - CouchDB offers stats over HTTP and you can either get all of them or each one separately. The problem that you can face when getting all stats one by one is that the stats themselves are affected by those requests. So if I monitor HTTP 200 responses and 3 other stats, each time I get new values the HTTP 200 responses will go up by 4, same for request time in seconds.

So that's why I think my best option to use Metriks is to use it as a third party library. What I did this far was to run a server that collects the data in bulk and then each separate plugin is a client to that server. I'm wondering if that could be incorporated into Metriks architecture?

[kvz]

Hi @sbnedkov, pretty much all RRD settings are configurable now, I've updated the Readme to reflect that. Long story short you can now also have counters and different steps.

You're saying you won't need api access anymore if you can somehow cache CouchdDB stats accross plugins? For '1 run'?

Plugins can be any language, but I guess it would be trivial in bash already:

#!/usr/bin/env bash
cacheFile=/tmp/cached_couch_stats.json
# Delete cache if older than 50 seconds. Silently fail for cold cache
find "${cacheFile}" -mtime +50s -delete 2>/dev/null || true
# Only download new stats if cache is cleared
[ ! -f "${cacheFile}" ] && curl 'http://couchhost.com/stats.json' -kso "${cacheFile}"
# Process json found in "${cacheFile}" according to current plugin logic:
echo "# config->interval: 60"
echo "# graph->title: Connections"
json_parse < "${cacheFile}" |egrep '\["connections","[^"]*"\]'

If you add this to all your couche plugins they could share the stats. you could also make 1 include file and source and share the above logic.

Not sure if I want to add this to metriks core. However I would accept a PR with your couch plugins and add those the metriks' ./plugins folder. This would give me some more real-life things to work with. And if I gut the core in big refactorings, I can make your plugins change with it.

Let me know what you think. I still also want to expose metriks as a node api at some point though, I'd like to be able to grow rrds from any app without having to create endpoints and plugins for that.