Provide metrics exposition

[dhardy92]

Thumbor should provide some way to expose metrics on some /metrics endpoint for example

I don't know if tornado can handle this feature natively but something like https://github.com/Cue/scales would help ?

[heynemann]

It looks like it can. Whenever I get some time I'll check it out. I would love to have metrics built-in.

[heynemann]

I'm not sure thumbor should expose a /metrics enddpoint, being an imaging server. I'd love to be able to send statistics to some endpoint (statsd, graphite and the like). Those are better suited to track and show stats, IMHO.

[heynemann]

What do you think @rcrowe @dhardy92 ?

[rcrowe]

All for that!

A stats handler that can be set through the config file could always create the /metrics endpoint if it's really needed.

Is there any plan / thought on how to tackle this?

[dhardy92]

if I look around, many services expose metrics via http endpoint at least.

• elasticsearch
• mesos
• hadoop
  It is easily integrable with any solution of collection throw collectd agent for example.

[heynemann]

What I meant is that I don't think thumbor should be responsible for displaying the metrics Damien... There are MANY solutions out there that do it properly. It's VERY easy to add a /metrics endpoint in your webserver that ballances thumbor (nginx?), so I think we should focus our efforts in integrating with those solutions. Do you agree?

[rcrowe]

Hey @heynemann

Do you have any thoughts on how this might be tackled? I'm pretty new to python, but I'd be willing to try putting a pull-request together to keep things moving.

Just after some some direction so I don't waste my time & hopefully will help others with making a contribution.

[heynemann]

Well, I would go on about doing it similarly to other extensible structures in thumbor, like error handlers or logging. That would allow users to publish metrics to whatever service they feel like doing.

That service would then be available from the context (importer) and could be used with whatever metrics we want to support. Some that come to mind:

• Image request;
• Original Image Cache Miss;
• Original Image Cache Hit;
• Result Storage Cache Miss;
• Result Storage Cache Hit;
• Image per original format (how many originals were a jpg);
• Image per resulting format (how many results were served as webp).

The last one is particularly useful to understand the portion of browsers that are being served webp images.

Maybe @dhardy92 has other metrics he would like to have. ;)

[ciceroverneck]

👍
More suggestions metrics:

• average response time avg_rt
• running_time
• exceptions

[phoet]

would be really cool to get more metrics out of thumbor. we experimented with NewRelic to gain more insight, but it looks like tornado is not supported any longer. if you have any pointers for how we could easily get more data into our librato setup or any other helpful service, please let me know. we are especially interested in the time it takes to fetch the original images from the web. i guess that would be Original Image Cache Miss.

[heynemann]

It seems incredibly useful @phoet do you think you can help with a Pull Request to add it?

[heynemann]

Any news here?

[phoet]

Please have a look at this, its not battle tested though https://github.com/thumbor-community/librato

[masom]

Thumbor now supports pluggable metrics which allows anyone to supply an implementation for their favourite data collection system.