Better Data Insights. Show me what's increased and decreased, movers and shakers (beta)

[mattab]

Update: see follow up issue https://github.com/piwik/piwik/issues/5246

Most popular pages/keywords/etc. are popular every day, and it is hard to know what changed since the day before. What would be interesting would be to visualize:
- Which pages significantly increased/decreased?
- Which URLs links/SE keywords brought me significantly more/less traffic compared to last period?

This could help answer questions such as:
- Why did the website traffic suddenly decrease? Traffic from some Keywords/Referers significantly decreased, and some pages significantly were less seen.
- Similar with answering “Why did traffic suddenly increase?” Why are my Ecommerce orders growing so much, how do I know the overall positive contributors explaining our growth?

1. User interface updates for new reports & hints
   1. New report: Overall Movers and Shakers

This report would have to query in particular: page URL, page Titles, Keywords, websites, campaigns, maybe 8 reports, for current AND previous periods. Making 16 API calls should work fine and comparison would be pretty fast hopefully.

This report could be available as a widget showing the top 5 movers and shakers overall (increased keywords, pages, etc.).

1. Extend existing report: Highlight Movers and Shakers

Currently reports display “Label”, “Visits count” by default. There is a button to display 5 more columns and a view for Goal conversions metrics.

We could maybe improve at least the “basic” view showing only visits.

In this view we could for example display in a new column on the right, a green or red % increase, when there is a a major change (threshold set to +/- 10% for example). Otherwise leave blank. See also #291

1. Extend report: Display Movers & Shakers in all reports’ footer

Below the report, or above?, we could display these rows from the report that have significantly changed since yesterday.

We could put threshold of minimum visits impacted = 20
minimum % increase or decrease = 2% for example?

So it would not display minor changes that are not interesting.

They could be displayed on top of the report

```
Best countries
(green sign) India (50%) and Scotland (33%)
(red sign) France (-33%) and Germany (-44%)
|————-|————|
| Country | Visits |
|————-|————|
| India | 176 |
| Scotland| 54 |
| France | 23 |
| Germany | 23 |
[….]
Next >

```

1. Display evolution % next to all sparklines

Idea is to show the increase and decrease of Visits, Conversions, etc. basically next to each Sparkline. There is a proposal mockup at: http://issues.piwik.org/attachments/5711/sparklines.png

1. Implementation

To produce a report “Period VS Previous period” for each keyword, we would need to select all keywords from both periods (2 datatables), and compare both to extract significant increases/decreases.

The code would be optimized so that it returns in less than 2 seconds when calling 8 different APIs so comparing 16 datatables.

The algorithm will typically extract:
- items with a growth of N %
- new items (new keyword since yesterday) with at least M %
- Never show items with less than P visits
- By default, we should enable the display in the UI for only the very significant changes, with a high threshold of visits impacted. The feature should not bring noise if the changes are not that interesting.

New widget to show this great insights:
- New smart widget to show what’s changed in dashboard automatically based on last login date: #3250

1. Similar tickets
   - compare 2 periods: #291
   - show evolution for a given keyword/page: #534

[mattab]

see GA hack on http://www.juiceanalytics.com/writing/keyword-trends-google-analytics-greasemonkey/

[mattab]

To produce significant movers and shakers, the algorithm could maybe select "current month" dataset and compare it to yesterday if period=day, or to last week if period is week, for example to find out the items with an average very different from the number "this week" ?

See also relevant article: http://www.wired.com/magazine/2011/06/ff_feedbackloop/all/1

ALso interesting: http://www.juiceanalytics.com/writing/keyword-trends-google-analytics-greasemonkey/

    display_limit: This controls the maximum entries that each table will contain. This may be useful for large, sprawling sites.
    growth_tolerance: This is the percentage growth parameter. Changing it to .10, for example, will catch everything that has grown by 10%, as opposed to the default 50% and 20%, respectively.
    minimum_number_elements: This is a significance benchmark that can be used to limit what is displayed upon the screen. By default, only keywords with at least 10 elements are displayed upon the screen. Referring Sites does not have a minimum by default, but one can be set if desired.
    limit: Limit is more of an internal parameter that determines how many entries should be downloaded from Google in order to get the results that are visible here on the page. Lower the limit to increase speed. If the limit is set to a very high number, you will get the largest result set, but you will have to sit around for a while for the results to load. Since the results are downloaded ordered by volume, raising the limit from the default numbers will not actually give more significant results. You will simply get more of the smaller results, such as keywords with only 1 hit.
    look_back: This is a very important parameter. The script uses the date displayed upon your Google Analytics page to determine the full range that you want to consider in your results, but look_back determines how many of those days are used for the significance test. So, say the range you have displayed in Google is March 23  April 22 and your look_back is 7 days. The script will compare the average referrals for a given keyword from April 16-22 to the average from March 23-April 15, and will return the keyword only if the recent average is 20% higher than the rest of the time period. Thus, if you want to increase the total range of the data, change the dates on the actual webpage. Change look_back only if you want to change the period of significance.

See also #3032 Trends reports

[mattab]

In general, this feature has much potential. Maybe other developers have better ideas to surface interesting data from the raw reports. We could write a simple plugin that would use the raw data to show insights. Other developers could then implement their own algorithm very easily by using the piwik APIs. We could even do a blog post and invite developers in a contest to write algorithms?

We could have a system that gives out several data insights profiles:

• Negative/Positive Alerts when significant decreases in different categories: SEO (keywords, sites), Content (urls, downloads), Marketing (campaigns), Ecommerce (Products), Technical (Browsers, OS changes in bounce rate), Audience (returning vis, countries, mobile)
• Display Possible actions for the users:
  • Marketing: check campaigns with visits but high bounce rate/low conversions,
  • Content: highlight high views ecommerce products with low conversion rates, highlight top viewed pages with higher than avg exit rate...
  • Technical: show significant pages with slow and/or slower than avg loading times

See also KPIs E-commerce : quels indicateurs pour votre tableau de bord for inspirations on good metrics that matter (french)

[mattab]

• Idea: Add new mode: color in red / orange, or green + lighter for good and bad rows. Make it a clog option. Also display, left of cog, "Enable Colors"?
• when looking at daily reports, show best yearly items not found in daily
• Similarly mode that compares with last month to show "New Links" or "New keywords"

see also nice ideas in http://www.kaushik.net/avinash/six-visual-digital-marketing-analytics-strategic-solutions/

[mattab]

Consolidating milestones FTW

[tsteur]

In 3055a65: refs #5477 started to work on displaying Insights. Added a widget for the dashboard and a new Insights visualization. More to come... still early version so expect a lot of things to change but I am still happy about feedback

[mattab]

In 3f2916b: Enable Insights plugin for all beta testers in the coming weeks. Please send us your feedback about this new Magic Feature! refs #5477

[tsteur]

In b062cb7: refs #5477 added tests and fixed some bugs

[tsteur]

In 7714334: refs #5477 do not call Insights API in integration tests

[tsteur]

In 2a60ff3: refs #5477 fix tests

[tsteur]

In 8110e9d: refs #5477 added possibility to place a widget multiple times (in case you want to see the same wdget with different visualizations and/or different parameters like different filters to make it easier comparable), removed some controls and save parameter changes in dashboard

[tsteur]

In 727da1b: refs #5477 some bugfixes and started to work on insights widget

[tsteur]

In 4de7581: refs #5477 fix tests

[hpvd]

coool :-)

Many thanks for your work on this!!

two tiny comments for looking inside on the table sites:

• should the filter bar stay at the bottom? At the top near to the segment-editor may be also a great place which may prevent the need for scrolling up and down when working with segment editor and insight filter at the same time...
• same with switching between standard and insight view: bottom or top? one idea maybe something like tabs at the top so you could easily see where you are and switch between the two views...

[hpvd]

using green and red as font colour for - and + values is great.
Maybe one could visualized the differences even more by using some kind of bars directly in the table.
Put an rough concept as attachment.

[hpvd]

Attachment:
2014-03-03_12h01_50.png

[tsteur]

In 856a3e6: refs #5477 reducing memory usage and improving performance

[tsteur]

In cbe88c1: refs #5477 destroy all tables but the insights tables

[tsteur]

In 6e022a7: refs #5477 also delete the last table

[tsteur]

In 62939c1: refs #5477 do not flatten for now, brings massive performance improvement 10min > 1 sec as Referrers_getKeywords takes forever and also leads in memory exhausted when report is flattened

[tsteur]

In 6034c0e: refs #5477 fix tests

[tsteur]

In 23186d7: refs #5477 make sure text ellipsis works and label is always only one line. moved test fixture to insights plugin as demonstration for other plugin developers as it is very insights related anyway

[tsteur]

In 8601c09: refs #5477 some more adjustments to the widgets and added a less file

[hpvd]

please see:
#4799
"broken after update to 2.1 final"

because disabling the inside plugin fixes it...

[tsteur]

In c25f396: refs #5477 started to work on movers and shakers and lots of minor improvements

[tsteur]

In a9e8651: refs #5477 some refactorings to have better testable code, loads of tests and bugfixes

[tsteur]

In 4bb969c: refs #5477 added more tests and some bugfixes

[tsteur]

In 862e532: refs #5477 removed min impact control as it might be not clear what it is anyway and user can change the limit filter if he wants to see only entries with very high impact

[tsteur]

In 593dbc8: refs #5477 some UI tweaks

[tsteur]

In ad9ab5d: refs #5477 more tweaks to insights and movers and shakers and more tests

[tsteur]

In 4165176: refs #5477 some code improvements and tests, also some adjustments for sites having not many visits

[tsteur]

In fc6fe7f: refs #5477 make sure always a change of at least 2 visits is required. Added icon license information, will move the information later

[tsteur]

In aa40466: refs #5477 fix syntax error

[tsteur]

In d210180: refs #5477 ui tweaks and translations

[tsteur]

In 81c3786: refs #5477 moving translations to general en.json

[tsteur]

In 78e9614: refs #5477 average filter is no longer needed

[tsteur]

In 3710bb4: refs #5477 some more code tweaks and documentation

[tsteur]

In a4f6fc5: refs #5477 moved legalnotice info for insights icon

[tsteur]

In e96399a: refs #5477 added missing translations

[tsteur]

In 2e1cb7b: refs #5477 make sure all column in overview widget have same width

[mattab]

In 9af9ae3: Refs #5477 New good looking icon for data insights for Morpheus theme!

[mattab]

Attachment:
Movers and shakers.png

[mattab]

Attachment:
insights overview.png

[mattab]

Attachment:
Insights view of a report.png

[mattab]

First version of this feature is now available!

• Current movers and shakers screenshot: [[http://issues.piwik.org/attachments/5477/Movers%20and%20shakers.png]]
• Current Insights overview widget screenshot: http://issues.piwik.org/attachments/5477/insights%20overview.png
• On the Referrers > Websites widget, click on the Light bulb "Insight" icon, here is what it looks like: http://issues.piwik.org/attachments/5477/Insights%20view%20of%20a%20report.png

Well done Thomas for this work, it is a significant feature in terms of usefulness and potential of "insights". we are committed to improving it in the future, so please, report your feedback!

Leave us your feedback, ideas, suggestions in this ticket: #5246 Data Insights, Movers and shakers: improvements and suggestions

#5246