Lack of process_* metrics

[hynek]

Recently I’ve run into the problem that my processes don't have any process_ metrics although the platform (Ubuntu Trusty) hasn’t changed.

I’m not sure how to debug the problem, do you have any idea what could lead to this problem? They are not using the new multiprocessing feature.

[brian-brazil]

Are you doing anything that'd make /proc inaccessible?

[hynek]

Not that I’d know of.

They’re run using runit and inside envconsul. The latter is the only recent change I can think of. Other than that, it’s regular processes in an LXC container with a distinct user.

Are you saying that /proc problems are the only possible reason why those metrics might be missing?

[brian-brazil]

With the current implementation, yes. We read from /proc/self/stat.

[hynek]

I've added a

with open("/proc/self/stat", "rb") as f:
    print(f.read())¬

block to the app's startup and before calling generate_latest(core.REGISTRY) and in both cases I get a legit-looking output back.

I suppose it's too late at that point anyway because something goes wrong write registering metrics?

[hynek]

OK it was /proc but not the self thing (free -m stopped working too). Seems like procfs crashed and I needed to reboot the container.

It may be nice if the docs mentioned this error case? Or a log message a la “could not access XYZ, process metrics won't be collected”?

[brian-brazil]

I don't think we need to document that if kernel is hosed things will break. We don't log as that could get spammy in other circumstances.

[hynek]

OK. Maybe if someone runs into it, they find this ticket. :)