Graph experiment statistics

[nilbus]

I'm working on a statistics page that visualizes the results of the exercism/discussions#123 experiment. The specific things I had planned to measure for this study are comment quantity and length (a loose measure of quality). I'm making some decisions and would like your feedback, @kytrinyx.

1. Where this page should exist: I'm planning on taking over /stats. Right now, it redirects to the first track, but it instead would be a page that stands on its own that still links to each track on the left. Would you prefer leaving the /stats intact and instead using a subpage?
2. General vs specific: Is it important or helpful if this pages lives on past the duration of this experiment? If so, I might call it the stats "overview", include data for time periods beyond the life of the experiment, and add overall iteration count in addition to the review quantity and quality that this experiment needs. If not, I could keep this page a lot simpler and revert to its former behaviour after we're done, and I would name it the intrinsic motivation experiment page.
3. I'm introducing a new JS plotting library, Plotly.js. The big issue I had trying to use Chart.js v1 (used currently for existing stats) is that x-axis labels cannot be inferred from from dates automatically at regular increments. I also like Plotly's hover effect that shows the individual data points.
   I attempted to update Chart.js to v2 but found that it handles large numbers of labels on the x axis like we have very differently. Rather than squeezing them close together (which is mostly functional but difficult to read), it truncates the list, not displaying any that don't fit (screenshot of C# w/ 105 exercises). It seems like it would be nontrivial to get Chart.js updated to v2 for existing graphs.
   With this change, we will be loading 2 chart/graph libraries. Although I'd like there to be just 1, at present I'm not considering it a priority to redo all our existing stats graphs using Plotly.
   Since the vast majority of users will never look at stats and therefore never need to load this library, I think we should leave this out of the bundled app.js, include it only on the stats page, and load it from a CDN. This isn't the first library we're including this way (html5shiv, respond, and styleguide.js), so I feel comfortable doing that.
4. I'll require users to opt out of the experiment to view these statistics during the experiment.

[nilbus]

2. General vs specific: Is it important or helpful if this pages lives on past the duration of this experiment? If so, I might call it the stats "overview", include data for time periods beyond the life of the experiment, and add overall iteration count in addition to the review quantity and quality that this experiment needs. If not, I could keep this page a lot simpler and revert to its former behaviour after we're done, and I would name it the intrinsic motivation experiment page.

I think I'll start by naming this "Participation" stats, which can cover both this experiment period and long-term participation.

[nilbus]

Another question: what is the order of magnitude of the total comment count we have on Exercism now?

[kytrinyx]

I'm planning on taking over /stats.

Yeah, that works.

Is it important or helpful if this pages lives on past the duration of this experiment?

I think it might be useful long term—let's assume that for now and re-evaluate when the experiment is over.

I'm introducing a new JS plotting library, Plotly.js

That sounds good. If I recall correctly, I chose chart.js because it was the first one I came across. Or maybe it was because someone had used it for something already. I don't actually recall. The point being: it was not a particularly reasoned choice.

With this change, we will be loading 2 chart/graph libraries.

Totally fine.

I think we should leave this out of the bundled app.js, include it only on the stats page, and load it from a CDN.

Yeah, that makes sense in this case.

I'll require users to opt out of the experiment to view these statistics during the experiment.

Agreed, that makes the most sense.

what is the order of magnitude of the total comment count we have on Exercism now?

We've got about 300k comments at the moment.

[nilbus]

Thanks!

I'll do some performance testing with 1 million comments, which should cover us for a awhile.