Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Support Ping requests to report a better 'time on page' #2041

Open
mattab opened this Issue · 41 comments
@mattab
Owner

Currently, the last page view is set with a 'time on page' of zero seconds.

Instead, we could regularly ping the Piwik server with a request that notifies Piwik that the user is still on the website.

This request should not record new page view, only increase the visit_total_time counter.

Also, we should only throw this request when the user is actually looking at the page, not doing something else in some other tabs.

The risk with this feature is that instead of improving the report accuracy, we end up inflating numbers: a lot of users leave pages open in other tabs for hours and don't do anything about it. We should heavily test to ensure we never throw a request on a page where the user didn't interact with recently. Some interactions we can maybe detect: mouse click, mouse move, etc.

Of course, we should not make everything slow by adding handlers on mouse move, etc.

Also, once this feature is implemented, a new checkbox should be added to the Advanced section in the 'Tracking Code' admin page. Checking it will generate JS code that enables this feature.

@robocoder
Collaborator

(In [3788]) fixes #739 - addEventListeners() on mouseup and mousedown; good catch, Matt!

  • utf8_encode() - required to convert multibyte char to string of bytes; for unit testing, it needs to be a separate function

  • killFrame() - would be called after a trackPageView(); used to prevent one's site from being frame by third-parties; there are tradeoffs (as you point) out, so it is not enabled by default

refs #2041 - setHeartBeatTimer - uses setTimeout, so it happens only once (vs setInterval is recurring); only happens on the landing page; intended to be used to improve bounce rate report; changed it to send ping=1 in the request

@robocoder
Collaborator

I reworked setHeartBeatTimer() in [3830]. It now takes an initalPingDelay and recurringHeartBeatDelay as parameters. By default, this is disabled.

@robocoder
Collaborator

The initial ping request contains "ping=1". Use this to improve bounce rates and visit time.

Subsequent heartbeat requests (if recurringDelay != 0) contain "hb=1".

I'm going to defer adding event handlers as there are issues with propagating some events like keyclicks.

Mental notes to self:

  • if you only send a beat when there's activity within the last n seconds, then on average, you'll expect the total time to be overstated by n/2 secs
  • to offset/compensate, you can instead send a beat if there's been activity in the last n/2 seconds

  • if there's no activity for n secs, do you stop checking for activity or stop sending beats?

  • what if activity resumes again? how long is too long ago to to resume beats?
@mattab
Owner

how do you define 'activity'? Will the ping be sent if user is looking at another tab, another window, or even away from computer?

@robocoder
Collaborator

activity: mousemove, click, mouseup, mousedown, mousewheel, scroll, DOMMouseScroll, resize, keypress, keydown, keyup, focus, and blur

  • need this many because of browser differences an the possibility of interference from other eveny handlers
  • handler simply sets the lastActivity timestamp

Yes, ping would be sent. That's why there's a check for activity. If there's no activity, I propose to no longer check.

I'll commit my changes after some more testing.

@robocoder
Collaborator

(In [3838]) refs #2041 - setHeartBeatTimer() now takes the minimumVisitLength and the heartBeatPeriod (time between pings) as parameters; this is simpler than initial ping + recurring hb; requests contain "ping=1".

also updated to jslint 2011-02-03

@robocoder
Collaborator

(In [3839]) refs #2041 - rollback buggy n/2 logic

@robocoder
Collaborator

(In [3840]) refs #2041 - undo previous optimizaton to ping on first page of a new session

@robocoder
Collaborator

(In [3841]) refs #2041 - add guard

@robocoder
Collaborator

Client-side changes are done, so it would be nice to have the server-side change done in time for 1.2 but we can roll into the next milestone.

@mattab
Owner

How often are the activity tests done? every second? Just thinking of keeping JS performance overhead low with this new feature.

I wont be able to do this in time for 1.2, so it will go in 1.3. Is pinging disabled by default? thx

@robocoder
Collaborator

Yes, disabled by default; frequency is user-defined.

@mattab
Owner

Before enabling this by default I think we could try it on some heavy websites, that are using GA, other libraries (jquery eg.) and doing Javascript heavily. Then we could see if Piwik adds any unexpected performance overhead.

For example it would be nice to say: Piwik will query a few JS variables every second to assess if the visitor is active or not, but will not conflict with any other library or Javascript and should not affect performance at all.

What do you think? for example, have you tried it on a a running piwik loading the dashboard full of widgets.

@mattab
Owner

I think, disabled by default, but easy to enable, this would make it a really cool feautre. Also, should be made available via the 'advanced' javascript tag generator see #1845

@anonymous-piwik-user

Just wondering where we are on this. I'm in the process of switching several sites from GA to Piwik, and this feature is pretty important to me. Does 1.x mean it's going to be in an indefinite release between now and 2?

@mattab
Owner

Polrbear, yes currently not roadmapped for 1.7 but it might change

@anonymous-piwik-user

I'm interessed too by this feature. I have activate it on my website, but the presentation in the server side, isn't perfect (the ping is show as a new page visit).

I hope, that it can be made for 1.7.

Thanks.

@mattab
Owner

Another bug report: http://forum.piwik.org/read.php?2,83913,page=1#msg-84060

NOTE: This feature does not work currently

@mattab
Owner

Proof Reading the upcoming book about Piwik, it's really good.

it gives me an idea related to this feature:

  • we should add a new parameter at the trackPageView function level, that users could use to trigger events/pageviews that will allow to disable the default "lower bounce rate". When "pinging" the page, we don't want to lower bounce rate by default, other users might want to do this as well in some cases.

We could allow for this other addon: send a fake Pageview 15seconds after arrival: http://analytics.blogspot.com/2012/07/tracking-adjusted-bounce-rate-in-google.html - might require the ping query so it doesn't inflate Page view counts, but changes Bounce rates for all other metrics.

@mattab
Owner

Note from Anthon:

The original thinking was, fire a ping once the page has been open some minimum amount of time, and keep pinging as long as there's some mouse activity. Basically, the goal would be to report lower bounce rates (ie if the first and only page visited) and more accurate time on site for the last page visited on a site.

The Page Visibility and traceCallback changes means the ping isn't sent until a tab gains focus and has been open the minimum amount of time. Further pings require some activity while the tab is open.

We probably need a way to restart the heartbeat if the tab regains visibility. Also, the inactivity logic doesn't work with passive activities (eg streaming media), but I think could be addressed by adding an API for players. (Are you using event tracking for this?)

This obviously isn't feature complete, so if anyone has ideas on how this should work or could work better, feel free to share your thoughts.

@mattab
Owner

I'm very interested in this feature but unfortunately time is missing to implement it for now, unless someone jumps in and produces a patch we can use! Or if you can sponsor the feature please get in touch at sponsors at piwik.org

@kevinoid

Great feature, looking forward to it. (Just commenting to add myself to the CC list.)

@mattab
Owner

See also: #5208 Detect when a user is actively browsing or using the webpage, for more accurate Time on page

@johsin18

I would really love to have this feature. There are more and more single-page applications nowadays, aren't there? For them this feature is kind of mandatory.

@RazvanMihaiu

Is this feature done? It's almost 4 years old now.

Thank you

@fredrikekelund

Just wanted to +1 this, would be a really great feature for using Piwik with SPA's

@jangrewe

another +1 here ;-)

@ru108

Try Using the Page Visibility API?

Feature Chrome (Webkit) Firefox (Gecko) Internet Explorer Opera Safari (WebKit)
Basic support 13 -webkit 10 (10) -moz 10 12.10[*] 7
33 18 (18)

http://caniuse.com/#feat=pagevisibility

@ismay

+1

@IPL01

Has this feature been implemented now?

I get a lot of single page views and they all report back as 0s and 100% bounce rates! Not good.

I am not a coder so suggesting doing something myself with the code or using an API isn't really helpful. Is there any chance something can be implemented in an upcoming update?

Many thanks.

@mattab mattab modified the milestone: Short term, Mid term
@Glisse1

Yay, short-term milestone! +1 :)

@mattab mattab modified the milestone: Piwik 2.14.0, Short term
@mattab
Owner

moved into 2.14.0 milestone, as this is such a useful feature, let's get it done :+1:

@hpvd

as already referenced above (#5208) taking existing events into account should help detect if page is only still opened in background tab without a performance problem.
=> so maybe a two step approach is possible:
1. for sites with already using events and
2. the ones which do not use events,
to be select in settings?
=> otherwise we may double effort and server load (if site alreday uses lots of events)

Another thing: do we need a "timeout" option e.g. no one looks a website longer than 15min to keep statistics fine? or 5min after last event?

@hpvd

since the function
"active life on open page / no life on page"
could be not only be used for
1) "better time on page for last page" (goal of this ticket)

but could also greatly be used as function within other features like
2) "detect page is only opened in background" details in #5845
3) "do not count automatic visits from website watchers" #3556 and #5845
4) identify unknown robots
5) ...

=> what do you think to make it a general function within Piwik?

It could bundle all technologies to decide
"active life on open page / no life on page"

  • Via triggers from events
  • via additional mouse movement tracking
  • via additional scroll tracking
  • via ping
  • via visits from same visitor on other pages at same time
  • via automatic timeout

and simply made the result available for every feature which needs it?

Of course not all technologies have to be active at the same time and one could develop a rule set
if no ...is send/received.. try ...try ...if nothing do automatic timeout

=> should we open a new ticket for this general approach? just did it #7607

@masteranalyze

I think the best solution ,is to use : Advanced Content Tracking like with Universal Analytics.
For reference i think this is the best method,that can simple solve the problem,and be quickly added on next piwik release:
http://cutroni.com/blog/2014/02/12/advanced-content-tracking-with-universal-analytics/
Original article: http://cutroni.com/blog/2012/02/21/advanced-content-tracking-with-google-analytics-part-1/

here’s what we’re going to track:

How many people scroll
When a person starts to scroll
When a person reaches the end of an article (not the end of the page, but the end of the article or post area)
When a person reaches the botton of the page (the bottom of the HTML)
Which website visitors are scanning my articles and which are reading my articles

Think about the value here! We will be able to get an accurate measure of which articles are actually read. We can even see which articles are so engaging that visitors continue through the comments to the botto of the page. Very useful stuff.

Tracking Technique

All of the above can be tracked with Event Tracking. The concept is that we will fire events when certain actions happen. Specifically we’re going to fire events based on visitors scrolling down the page.

Finally :+1:
Increase in the Average Session Duration. Events also increase the Time on Page measurement on exit pages. This may cause an increase in the Average Session Duration metric.

Increasing Bounce Rate is not Necessarily a Bad Thing!

If a visitor comes to you site, reads a post in-depth, and then leaves, should they really be classified as a bounce? Probably not. With this implementation, your bounce-rate will provide a more accurate measure of engagement and it will solve the real time spend on the website!

I hardly wait for this implemenation to be alive!

@masteranalyze

Except this events,in order to even have an more accurate time it can be added:

1)Track user that print pages :
Here is the code for google analytics that can be easily converted to piwik to track this events:
https://www.savio.no/analytics/how-to-track-printed-pages-in-google-analytics

2)Track Downloads & Other External Links,track mail to,telephone clicks.
Here is the code for google analytics that can be easily converted to piwik to track this events:
http://www.blastam.com/blog/index.php/2013/09/howto-track-downloads-links-universalanalytics

3)Social actions on page engagement:
Here is the code for google analytics that can be easily converted to piwik to track this events:
http://www.optimizesmart.com/social-interactions-tracking-through-google-analytics/

It`s working for majors:Twitter, Facebook, Google Plus & Linkedin,others can be added as need it.Maybe pininterest,stumble,etc.

If we track the 5 events from the cutroni example+3 that adds up here like :print,download,external links,and social interaction,we will be able to get an more better ideea of the interaction on that page,and not only the time will be more accurate,it will be the bounce rate too more accurate,as that user has done action and interacted with that webpage,so it should not be consider it an bounce as the user interacted with that webpage by sharing,mail to,phone,read,etc.

I think this can be implemeneted right the way with the help of the events from piwik.

@mattab
Owner

Not only have many people commented on this issue but in the forums this ticket was linked to about 40 times in the last few years.

@Glisse1

Just wanted to share the forum post from a year ago with suggestion as of what direction the visual displaying of real-time status (active, idle, reading, writing, etc) could be going... knowing the true active time on page in statistics is important, but I believe it`s just as important is to have some sort of visual representation in real-time widget

http://forum.piwik.org/read.php?3,112235,page=1#msg-112235

@masteranalyze

"Also, once this feature is implemented, a new checkbox should be added to the Advanced section in the 'Tracking Code' admin page. Checking it will generate JS code that enables this feature. "

Exactly ,this is how it should be for ping in,and for the events method,so users can simple enable from events,if they wish to enable : bounce handle,social network sharing,print,download,etc.
and once enabled,the code to automatically track them,to not have like now,to add alot of scripts for those events.
For example this:

script
setTimeout("_paq.push(['trackEvent', 'time spent', '>15sec', 'good reading'])",15000);
/script
This will make more simple to enable events and users on page time will improve and we will know it real,and also the bounce rate will be adjusted.
By solving this we will solve not only the 1st page timer ,but also the real bouncing rate,as from my point of view if an user spend 15-30 seconds or even more on just that webpage and he is interacting:sharing,printing,downloading or any action that can be done into an webpage,that is not an bounce at all.
That means the user find very usefully that blog post or whatever,and we cannot now that,unless this feature will be ready on 2.14.
Hopefully it will be alive!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.