Page Overlay - Analytics Overlay on your site! (Aka in-site analytics)

[mattab]

In order to show analytics data as an overlay, we have to set up a cross-domain communication protocol. The aim of this ticket is to complete a simple version of the feature that provides extensible mechanisms for the communication. Based on this, we can add more things to it.

Page Overlay displays a block around each link showing the % of clicks on this link. the user can still browse the website. On the left a sidebar displays the key statistics for the currently viewed page. It also easily allows the user to open the Page Transitions report and the Row evolution historical report for this particular page.

See #3530 List of Page Overlay Improvements for more info and the current state of the art!

[mattab]

From EZDesign:

I did some research on in-page analytics. Sounds very interesting.

But as it turns out, manipulating the iframe is far from easy to do. I think, I figured out how Google does it and posted it as a reply to a stack overflow question:

http://stackoverflow.com/questions/5210897/how-does-google-analytics-in-page-analytics-work/

I think it's quite problematic, that Google injects something the user doesn't really want, which demonstrates what kind of power Google has. I think this is not really a problem in our case, because we don't inject it, the users do (from their servers which are under their control).

In general, is this feature something we want? It definately is a piece of work, but it's quite a fancy feature. Seeing how much people like to be close to their visitors (i.e. Live), this should be received well!

[mattab]

Timo, in page analytics would be a great feature to have for sure... especially since we can do it with the current data set and don't need to track more data.

The main limitation of this feature I think is that it doesn't differentiate few places linking to the same URL, so you don't know if users clicked mostly on top link or bottom link to "Contact us". But I guess we could somehow fix this issue later on, by storing some kind of unique ID for the DOM node being clicked, and add this field in the log_link_visit_action table ;) this would be hard code but I think possible without changing too many things. Anyway, not for V1!!

We probably could not pre-process the data, so it would be read from the logs directly (still fetched from an API function). Performance wise, we don't have an INDEX on the field idaction_url_ref so GROUP BY on this field will be pretty slow. We could add the INDEX, or maybe just "recommend" users to do it if they find the feature slow to use...? Adding the index for everybody sounds not ideal since it would have a non trivial performance/disk space overhead. But, it might work fine without it for most users anyway!

Also there will be some interesting DOM manipulation to detect all links to another URL (and normalizing URLs in the page transforming ../ etc. in the full URL).

Note that in Piwik we record the URL after removing some URL parameters. You can call getSiteFromId() and then look at excluded_parameters attribute to have the list of all excluded parameters. It is important to remove these from the links detected on the page to ensure you can match the same URLs. Also, common session parameters are excluded (see API SitesManager.getExcludedQueryParametersGlobal()) as well as Piwik campaign parameters (see campaign_var_name and campaign_keyword_var_name in global.ini.php).

Note that I have tried GA in page analytics on 3 different GA accounts but it didn't work in any of them... so I feel like the other guy in stackoverflow "The answer to 'how does it work' is 'it doesnt work'" ;)

Regarding how it works, I haven't looked into it but my first idea would not be to do iframe overlay, but to inject additional DOM elements that would show up on Hover and show the data for each link... Not sure if that would work or if iframe idea is better?

I think that, if we implement this feature, we'll need the "filter" dropdown as well, without it the page would look too busy and full of data maybe..

It would be great & useful & fun feature :)

PS: see also: https://www.google.com/search?q=webtrends+smartview

[timo-bes]

The current working title of the feature is "Insight" or "Page Insight". We recently started using the word "Insight" for intelligent alerts, which would probably be a better way to use it. So we might want to come up with a better name.

Others call the feature "in-site analytics" or "on-site analytics". One suggestion I have is "Piwik Overlay" - but in the menu, it would be shortened as "Overlay". Or maybe "Page Overlay"?

Every comment is appreciated.

[timo-bes]

(In [7223]) refs #2465: Starting the Insight plugin (in-site analytics)

This is the code I wrote about a year ago :)

There are several things that will be changed / removed but it's still good to have it in SVN - that's why I commit it like this.

[timo-bes]

(In [7224]) refs #2465: removed code for resolutions overlay - it turned out to be not that useful. maybe we can rethink it in the future.

[timo-bes]

(In [7225]) refs #2465 svn properties for new files

[timo-bes]

(In [7226]) refs #2465 removed alias domain normalization. it's now consistent with the behavior of transitions which is good for reusing the transitions report.

[timo-bes]

(In [7227]) refs #2465, refs #3332: using Transitions for getting the following pages in Insight

[timo-bes]

(In [7228]) refs #2465 Insight is starting to be cool

• highlight links on hover, displaying a border and the absolute number of clicks. also, the bubble (along with all other bubbles for links to the same url) is highlighted and the url is shown in the status (lower right corner)
• bubbles are updated when links are hidden or added. this is useful for drop-down navigations or ajax content.
• code formatter applied (some whitespace changes)

[mattab]

Just an idea while I think of it:

• it would be great to also show a link to open Transitions popover. I will review in more details later.
• also would be great to see a link to Row Evolution, as it shows historical evolution
  Maybe these 2 could be next to the 2 icons, written in grey, below the list of Metrics, in a new section "More insights on this Page:" ?

[timo-bes]

(In [7261]) refs #2465 improved efficiency of insight tracker extension.

you can review piwik.js now, i don't plan to change it any more.

[timo-bes]

When you open an insight session, you get JS errors like this (at least in Chrome, I haven't checked other browsers):

Unsafe JavaScript attempt to access frame with URL http://piwik.server/index.php?module=CoreHome&action=index&idSite=1&period=week&date=2012-06-24#module=Insight&action=index&idSite=1&period=week&date=2012-06-24 from frame with URL http://web.site/path/. Domains, protocols and ports must match.

It is thrown in piwik.js / getReferrer (line 640) because Piwik is checking the referrer of the parent document, which is the Piwik UI. The same origin policy doesn't allow that.

Who knows how we can avoid that?

[timo-bes]

(In [7262]) refs #2465 using piwikHelper.ajaxCall to build ajax call. it now works for date ranges as well.

[timo-bes]

(In [7263]) refs #2465

• FULL SCREEN MODE!
• smoother handling of the case when there are no pageviews for the current url
• hovering the bubbles highlights the links as well
• safer repositioning of bubbles (timeout instead of interval)
• adding breakpoints to the current url

[mattab]

(In [7306]) Refs #2465
Fixing JSLINT errors + packaging changes in piwik.js in the minified script

Note there are still 2 JSlint errors.

PS: Timo, I will carefully review your work on this ticket after 1.9.1 is out!

[mattab]

This ticket will ideally be closed on Nov 15th, then we could reopen a new one with the pending new features and ideas for improvements.

[timo-bes]

(In [7392]) refs #2465

• removed hardcoded domain (this is what caused the error on the demo and all other piwik instances)
• force iframe height in firefox (height:100% doesn't work)

[timo-bes]

(In [7393]) refs #2465

• this time, forcing the iframe height is included in the commit
• client side url normalization: remove excluded parameters

[timo-bes]

(In [7405]) refs #2465 Insight

• links to the current url get a bubble as well
• limit for the number of pages loaded from the API can be configured
• ready for translation
• url normalizer doesn't remove the hash part anymore (which corresponds to the latest behavior of the tracker)

[timo-bes]

Here are some things I'd like to hear your comments about:

• As mentioned before, the name Insight would be better for intelligent analysis and alerts. Further, "Overlay" might be a better name for this plugin. How do you feel about renaming it Overlay? This would also be more suitiable in combination with heatmap.
• The data quality in Transitions and Insight would be improved a lot if the tracker would not record outlinks and downloads as previous actions. When a visitor clicks multiple download links on one page, only the first one will show up in Transitions and Insight because for the other clicks, the previous action was the download made before. Would it be a good idea (and doable) to track this differently - i.e. when a download/outlink occurs after a pageview, keep the pageview as the previous action for the next download/outlink?
• There are commented out test cases in client/urlnormalizer.js. Obviously, that's not the right place for them. But where would be the right place?

[timo-bes]

Could you please release a beta so I can do some more tests?

[timo-bes]

(In [7418]) refs #2465 Insight

• new action: open in new tab - opens Insight session without an iframe
• handle excluded parameters when loading the sidebar and following pages data
• highlighting of linked images works properly now
• tooltip showing the domain of the iframe

[timo-bes]

(In [7419]) refs #2465

• renaming menu item to "Page Overlay", putting it under Actions
• config option to disable framed mode altogether (can be used when websites contain frame busters)
• fixed error for anonymous user
• loading message in framed mode

[timo-bes]

In Piwik_Insight_API::getExcludedQueryParameters I had to catch an exception that is thrown in Piwik_SitesManager_API::getExcludedQueryParametersGlobal when the user has no admin access at all (Piwik::checkUserHasSomeAdminAccess). In this case, excluded parameters are not handled by Insight. This means that when using anonymous access, excluded parameters don't work.

Wouldn't it better to check in Piwik_SitesManager_API::getExcludedQueryParametersGlobal whether the user has some view access instad of some admin access? Or what's the reason for checking admin access?

[timo-bes]

(In [7420]) refs #2465

• links in the pages report can be opened in insight instead of a new tab (to enable, use config option insight_launch_from_pages_report)
• links containing an image should really work properly now
• plugin description set

[mattab]

(In [7426]) Testing Page Overlay on demo Refs #2465

[mattab]

• As mentioned before, the name Insight would be better for intelligent analysis and alerts. Further, "Overlay" might be a better name for this plugin. How do you feel about renaming it Overlay? This would also be more suitiable in combination with heatmap.

Overlay sounds like a plan indeed. It is more general and will allow us to easily include other concepts in it. Do you mind renaming all files and the plugin itself? It's quite a job but definitely worth it. we are regretting not doing it for PDFReports which should be called ScheduledReports.

• The data quality in Transitions and Insight would be improved a lot if the tracker would not record outlinks and downloads as previous actions. When a visitor clicks multiple download links on one page, only the first one will show up in Transitions and Insight because for the other clicks, the previous action was the download made before. Would it be a good idea (and doable) to track this differently - i.e. when a download/outlink occurs after a pageview, keep the pageview as the previous action for the next download/outlink?

it's great data value to know, from which page URL downloads were made. Because I'd like to know, where latest.Zip was downloaded from exactly. This is why it would be useful to be able to click on a File or Outlink in Transitions, to get the referrers for this download/outlink.

But you are right, that we should not track referrers when they are downloads or outlinks as it does not make as much sense!! Here is my proposed patch which I haven't tested (tests take still forever for me, yet to get a SSD ;)) but it will work I think:

+++ core/Tracker/Visit.php  (revision )
@@ -349,7 +349,13 @@
        $sqlActionUpdate = '';
        if($idActionUrl !== false)
        {
+           // In Transitions and Site Overlay reports,
+           // A given Action being looked at can only have an internal referrer of Pageview or Search.
+           // Downloads and Outlinks will not be tracked as "previous action".
+           if( in_array($actionType, array(Piwik_Tracker_Action::TYPE_ACTION_URL, Piwik_Tracker_Action::TYPE_SITE_SEARCH)) )
+           {
-           $valuesToUpdate['visit_exit_idaction_url'] = $idActionUrl;
+               $valuesToUpdate['visit_exit_idaction_url'] = $idActionUrl;
+           }
            $sqlActionUpdate .= "visit_total_actions = visit_total_actions + 1, ";
        }
        if(!empty($idActionName))

EDIT: we should remove sum_time_spent on all outlinks and downloads Datatables and expected/ API outputs since it currently contains sum_time_spent, and when this patch is applied they all become zero.

• There are commented out test cases in client/urlnormalizer.js. Obviously, that's not the right place for them. But where would be the right place?

We have javascript tests running in: piwik/tests/javascript/

See: https://github.com/piwik/piwik/blob/master/tests/javascript/index.php#L425

You can create test groups, and easily add powerful unit tests to all the new Javascript code. If you read through the file once you should get a good idea of how tests work. I have tried to maintain the tests coverage high of piwik.js since Anthon's initial great work on these JS tests. +1 to add solid unit tests coverage for the tricky URL normalization parts!

[mattab]

Timo, you did it again!! An awesome new feature designed in a very nice fashion... I can't stress enough how great it feels :-)

I will put the review first of the UI and some suggestions for changes and tweaks to make UI better and more usable - I will post code review in a few hours.

UI/UX

• I like the "Page: /XXXX" at the top of the iframe. However it takes up one line which removes pixels for the initial view of Transitions feature. I think the report would look much nicer, if the "Page Name" being looked at was actually displayed inside a new BOX similar to the "Calendar" that would read "Page: /XXXX" similarly but this way, it would not take an extra line.
  • The box would also highlight the Page URL being looked at and would clarify what this is about. Even maybe it should say "Page Overlay for: /XXX" ?
• In the left sidebar:
  • it would be nice if "View Transitions" and "View Page Evolution" were available with icon+text similar to Full Screen link, and would open the Row evolution/Transitions report as an overlay on top of the Site Overlay. This would be Uber Awesome.
  • "Open in new tab" can use same CSS style as "Go Full screen" and use the std piwik orange-link icon
• I Love the full screen mode, and the Awesome display on overlay on the website with no extra parameter!
  • is it possible to ask a new mode: Trigger the overlay reports by calling http://example.org/#overlay (and default to the "User default" date + period, see ), without requiring to have Referrer?
  • Could we display a half transparent Box that would invite user to click, which would then expand into an infobox, overlaid on the website (Maybe at bottom left or right?), that would show: the date range being looked at, and the list of metrics.
  • This overlay box in the full screen mode, could later contain the calendar to let the user change the date that was pre-selected when using #overlay
  • In general, the calendar should be displayed in the sidebar in the "Full screen mode" so that users always see which period the report is applied for. This is quite key in Piwik to always leave calendar or date displayed (like in Transitions)

Bugs Reports

• On demo it does not show the number of clicks on the download file http://piwik.org/latest.zip - However, these clicks are shown correctly in the Transitions report so the underlying data is OK. It's important to see clicks to files in the overlay repotr since this is an excellent feature of Piwik VS GA because we do automatic file tracking :)
  • Same for outlinks. Would be nice to see how many click on these buttons to fb/twitter for example (and could be very useful for checking advertising campaign queries)
  • this is why I think the #overlay is very important. A webmaster could simply check the effectiveness of the marketing campaign from one page by going to the page and type #overlay and start his piwik experience from there!
• when I have firebug enabled, the Iframe becomes much smaller (see attached screen). It would be nice if instead the iframe was the normal height with firebug on
• The overlay for piwik.org Language selector menu bubbles are displayed, for items that are hidden. It would be nice if the hidden Javascript menu bubbles were displayed only when user hovers on the menu inside the Page Overlay Piwik report. The bubbles are currently always displayed.
• Loading Page Overlay while using Piwik over SSL does not work in: https://demo.piwik.org/index.php?module=CoreHome&action=index&idSite=3&period=year&date=2012-11-08#module=Insight&action=index&idSite=3&period=year&date=2012-11-08
  in Chrome: Unsafe JavaScript attempt to access frame with URL https://demo.piwik.org/index.php?module=CoreHome&action=index&idSite=7&period=year&date=2012-11-08#module=Insight&action=index&idSite=7&period=year&date=2012-11-08 from frame with URL http://forum.piwik.org/. Domains, protocols and ports must match.
  • When Piwk is used over SSL the Page Overlay should load the site in iframe over SSL.
• Data accuracy: as suggested by email, I think Form Post url discovery would be very nice to have!
• When the page does not load for any reason, could you display a NOTICE message (in a JS Alert box or in the screen) such as "Page Overlay Report could not load for this website. here short troubleshooting or link to FAQ?"
  • If we detect that Piwik is currently loaded over SSL, and the site doesn't load, we can suggest that the site does not support HTTPS and suggest them to use Piwik over http to view site overlay?

[mattab]

Attachment:
overlay - small height when firebug enabled.png

[mattab]

(In [7428]) Refs #2465 - Only require some view access to get excluded parameters
As per Timo's suggestion in #2465

please remove the exception handler

[mattab]

(In [7467]) Refs #2465 Will test on demo

[timo-bes]

• Message can be slightly improved:
  • If the user is admin for the site Piwik::isUserHasAdminAccess($idSite) then the message could link to: module=SitesManager&action=index with hash tag: #row5 for idsite=5 --> this way they can directly edit the settings and we make the feature pain free!

Great idea! The only problem is that #row5 links to fifth row, not the row with idSite=5. So how could else could we link to right row in the table?

[timo-bes]

(In [7470]) refs #2465

overlay has two modes now: full screen with sidebar & full screen without sidebar. for now, it is only accessible via a new row action in the pages report.

because of this change, some mostly not-so-nice core-changes could be removed (mainly that's good because overlay doesn't impact as many things outside the plugin anymore):

• disableHistory option of broadcast
• handling for parameter overlayUrl in broadcast
• openLinksInOverlay() method of html table
• config option overlay_launch_from_pages_report
• hack for data table to launch overlay from the links

other changes:

• error message for wrong domain has nice styles and links to admin page when the user has admin privileges
• when registering row actions, the order can be set
• made row action style a little more compact
• increased min width of label column a bit

[timo-bes]

(In [7471]) refs #2465

• simple date selection for sidebar mode
• code formatter applied (whitespace changes)

[timo-bes]

(In [7472]) refs #2465 iframe dimension fix

[timo-bes]

(In [7473]) refs #2465

• row evolution and transitions can be launched from the sidebar
• when launching overlay, always pick date "current month"
• make sure there never are two scrollbars when the window is not high enough

[timo-bes]

(In [7477]) refs #2465

• remember the location of the iframe in the l parameter using the core broadcast methods => popoverParam works as well => transitions and row evolution can make use of the back button
• row evolution and transitions get a static launch method that is used in overlay to trigger the popovers without any knowledge of their implementation
• broadcast.propagateAjax() gets the disableHistory parameter back but the handling is much easier this time
• remove full screen link for now because we might want to do that differently - i'll add some details to the ticket later
• two fixes for sanitized urls

[timo-bes]

(In [7479]) refs #2465

• support for internet explorer 7 & 8
  • redirect by setting window.location.href loses referrer - use fake link
  • implementing position:fixed for status bar in java script
  • when visibility=inherit, go up the dom to find out whether an element is visible
  • avoid .data() completely in followingpages.js
  • some specific html and css
• adding v parameter to loaded script to invalidate caches
• regenerating piwik.js (i hope it worked properly this time)
• hiding notifications that get stuck after removing a hovered element (only happens sometimes)
• binding window.resize after every broadcast.pageload() - the binding was lost after opening a popover
• fixing mode with frames disabled: didn't work after migration to the l parameter

This version is the first one that I think is stable. Please deploy on the demo so we can do some more testing.

The next step is translation and documentation. After that, we're ready to release as beta.

[timo-bes]

(In [7482]) refs #2465 link new doc page from overlay error message

[mattab]

(In [7483]) Refs #2465 Will test on demo

[timo-bes]

(In [7486]) refs #2465: using Piwik::isHttps()

[mattab]

(In [7493]) Refs #2465
FIxing " The only problem is that #row5 links to fifth row, not the row with idSite=5"

[mattab]

(In [7516]) Refs #2465 Will test on demo

[mattab]

(In [7519]) Refs #2465 fixing error - but i'm afraid it still not loading on https page on demo.

[mattab]

(In [7528]) Refs #2465 Will test on demo - this beta should be fairly stable!

[mattab]

(In [7534]) Fixes #3572 Refs #2465 Overlay work on SSL on demo! NICE!

[mattab]

See #3530 List of Page Overlay Improvements for more info and the current state of the art!

[halfdan]

I have a bug (?) report for Page Overlay, sadly Trac blocked my post so I reposted it on pastebin: http://pastebin.com/DtwLvPXi

[mattab]

Error is: Strict Notice:</strong> <i>Only variables should be passed by reference</i> in <b>/var/www/htdocs/piwik.c0demonkey.com/html/plugins/Overlay/API.php</b> on line <b>116</b>

[mattab]

(In [7712]) Refs #2465 Try fixing notice - does it fix it @halfdan?

[mattab]

(In [7713]) Refs #2465 that's better I think

[mattab]

Marking as fixed. Nice work Timo! Hope to see some improvements in Page Overlay later!