Skip to content
This repository

Zotonic statistics #508

Open
mmzeeman opened this Issue January 26, 2013 · 9 comments

2 participants

Maas-Maarten Zeeman Andreas Stenius
Maas-Maarten Zeeman
Collaborator

We currently don't have any statistics. Let's add it. The zynamo uses statz and has an adapted webzmachine which collects statistics.

How do we want to implement this for master? Use folsom? Which has two deps, bear and meck (for testing), or use statz?

Folsom can be used and can be hooked into zotonic at the same spots as statz.

Apart from collecting the stats, we also should make them visible somehow. Any thoughts for this? (I kind of like http://g.raphaeljs.com/ but if there are other options)

Andreas Stenius
Owner

Perhaps have generic collection points (using notifications) and custom modules to collect them into one of any possible stats collectors (such as statz, folsom, graphite, or other), where graphite has it's own support for presentation.
Personally, I'd like to use folsom and the g.raphaeljs looks neat :)

Maas-Maarten Zeeman
Collaborator

Hmm, yes. Would you like to be able to turn off stats collection? Or plug in another notification system?

Some of the collection is done in the core of webzmachine, which can't use notifications.

Side thought, maybe someone also wants to collect stats on the notifications too.

Andreas Stenius
Owner

Ah, good point.

Turn off - well maybe.. ? But that may not be needed at the collection points.
Better avoid the notification system, I guess it's better to simply send notifications directly to the a stats collecting process. Collecting stats on the notifications would be nice, too :)

I think the collecting should be fixed, but the process receiving the stats and processing them should be pluggable.

Maas-Maarten Zeeman
Collaborator

Found another statistics collecting library which looks quite nice. git://github.com/knutin/statman.git

Maas-Maarten Zeeman
Collaborator

After thinking it over a little bit I would like to propose the following.

Collect the stats via a set of bride modules named z_counter, z_histogram and so on. The modules are provided in a separate erlang app. The bridge modules know about zotonic so it knows that it has to collect statistics for the whole of zotonic as well as for each specific site. The actual implementation of the bridge modules depend on which stat collection library is chosen.

One thing I am not so sure about is how we can make statistic collection optional.

1) We can call catch z_counter:incr(Host, db, requests) instead of calling it directly. This also makes sure that collecting statistics never breaks something on the critical path.
2) Instead of using anti-erlang-ish pokemon exception handling we could also create a dummy set of bridge modules which are just contain no-ops.

I want to start with a collection app which uses git://github.com/knutin/statman.git

Andreas Stenius
Owner

Why not use the standard OTP event handler architecture. That way, zotonic defines the events and the event manager, and client apps collecting stats for various end uses can hook into the event manager without zotonic being affected either way.

Maas-Maarten Zeeman
Collaborator

That will be a busy event manager. I'm a bit worried about that it becomes a bottleneck pretty quickly. But +1 for the defined update events. I like that idea.

Something like this?

z_stats:update(#counter{name=requests, host=Host, sub_system=db})
...
z_stats:update(#duration{name=duration, start_time=Start, host=Host, sub_system=webzmachine})
Andreas Stenius
Owner

Who ever that will act as the sink of those events are going to be busy. The question as I see it is how to connect the source to the sink. Sure enough, if the event manager can't keep up and his message queue builds up, it's going to affect the event sources sending the updates, but in that case, maybe the manager ought to run on a dedicated machine, so the events are sent off-machine as early as possible.
Or do you believe that the event manager adds a lot of extra overhead? (I hope not, but don't know).

The update api seems good, but the #duration{} has a start_time, but no end_time nor duration.. ?

Andreas Stenius
Owner

I had a little chat with @mmzeeman here http://zotonic.com/chatlogs/2013/02/04.html
So I'll post our conclusion from that here for the record:

To avoid as much overhead as possible while still being flexible towards selected stats backend (statman, folsom, graphite, et al), we think that a single config value in the app environment naming a bridge module responsible for translating zotonic stats events and forwarding them to the appointed stats backend would be enough.

As the config is in the app env, it is accessible without a zotonic context, it can be read during startup to be kept at a convenient and efficient place for later lookups.

As it were, it feels like a dedicated stats context may be the best solution. Currently only the backend module config is needed in this context, but if we define it as a record from the start, we can easily add more stuff in here in the future without breaking a lot of existing code.

cc: @arjan @mworrell any input on this? I think Maas will get going with this soonish... ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.