/proc/stat: absolute values vs normalization by USER_HZ

[cournape]

Would you consider reporting the cpu-related info from /proc/stat in ticks / sysconf(_SC_CLK_TCK) instead of ticks ? It would make comparison between VMs and across kernel more robust ?

[juliusv]

@cournape This is already happening, no? See https://github.com/prometheus/node_exporter/blob/master/collector/stat.go#L115

We will probably get rid of that sysconf call soon though, because it needs cgo, which doesn't work with cross-compilation. We have done some research and it seems USER_HZ is 100 for all platforms that Go is available for, so it can probably just be hardcoded.

[cournape]

Hm, not sure why I thought it wasn't, I will have to check the data again (I got the impression from the values itself, I did not look at the code, mea culpa).

Some people change USER_HZ when compiling their kernel, and I think it is arch dependent, though that may be moot for an application written in go.

On the topic of cpu stat, would adding features on a per-process basis (based on /proc/[pid]/stat and /proc/[pid]/statm) be on topic ? I am implementing a PoC using python, but I figured that may be interesting to other node exporter users. Processes to be tracked would be defined in terms of pidfile, or /proc/[pid]/cmdline regex, etc...

[brian-brazil]

On the topic of cpu stat, would adding features on a per-process basis (based on /proc/[pid]/stat and /proc/[pid]/statm) be on topic ?

We'd generally encourage instrumenting individual applications with prometheus and getting not just the cpu/memory stats but also other stats relevant to your services. Having such information tied to the machine rather than the service makes monitoring much trickier, and node_exporter is fundamentally about the machine.

[juliusv]

Note also that when instrumenting any application via the Go client library (perhaps others already as well?), you automatically get CPU and other runtime stats included by default in that application's /metrics endpoint.

[cournape]

In some cases, per-service monitoring is not always practical/feasable (e.g. reverse proxies, DB). But I agree that when possible in-app monitoring is more tractable.

[brian-brazil]

We have the haproxy_exporter, and I believe someone's working on the various relational databases.

[juliusv]

Yeah. @cournape So as a guideline, we want to keep the node exporter all about things that are really tied to a machine, and keep service-specific metrics out of it (even if it's just CPU stats etc. about those services). Exceptions are for services that are really, really closely tied to a machine (like the gmond collector module). Note that you can still always export anything you want from the node exporter (if needed) using the textfile collector module.

[cournape]

So the idea is to have this kind of monitoring split in separate services. Makes sense.

[juliusv]

Yep. Regarding USER_HZ, AFAIK it's not configurable (only CONFIG_HZ is). Do you mean they actually change the kernel sources before compiling?

If we ignore that case, it seems there are only 2 exceptions to USER_HZ == 100: http://lxr.free-electrons.com/ident?i=USER_HZ

ia64: configurable (it matches CONFIG_HZ)
alpha: 1024
all the rest: hardcoded to 100

[juliusv]

(and ia64 and alpha are not platforms for which Go is available)