Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Support Ping requests to report a better 'time on page' #2041

Closed
mattab opened this Issue · 53 comments
@mattab
Owner

Currently, the last page view is set with a 'time on page' of zero seconds.

Instead, we could regularly ping the Piwik server with a request that notifies Piwik that the user is still on the website.

This request should not record new page view, only increase the visit_total_time counter.

Also, we should only throw this request when the user is actually looking at the page, not doing something else in some other tabs.

The risk with this feature is that instead of improving the report accuracy, we end up inflating numbers: a lot of users leave pages open in other tabs for hours and don't do anything about it. We should heavily test to ensure we never throw a request on a page where the user didn't interact with recently. Some interactions we can maybe detect: mouse click, mouse move, etc.

Of course, we should not make everything slow by adding handlers on mouse move, etc.

Also, once this feature is implemented, a new checkbox should be added to the Advanced section in the 'Tracking Code' admin page. Checking it will generate JS code that enables this feature.

@robocoder
Collaborator

(In [3788]) fixes #739 - addEventListeners() on mouseup and mousedown; good catch, Matt!

  • utf8_encode() - required to convert multibyte char to string of bytes; for unit testing, it needs to be a separate function

  • killFrame() - would be called after a trackPageView(); used to prevent one's site from being frame by third-parties; there are tradeoffs (as you point) out, so it is not enabled by default

refs #2041 - setHeartBeatTimer - uses setTimeout, so it happens only once (vs setInterval is recurring); only happens on the landing page; intended to be used to improve bounce rate report; changed it to send ping=1 in the request

@robocoder
Collaborator

I reworked setHeartBeatTimer() in [3830]. It now takes an initalPingDelay and recurringHeartBeatDelay as parameters. By default, this is disabled.

@robocoder
Collaborator

The initial ping request contains "ping=1". Use this to improve bounce rates and visit time.

Subsequent heartbeat requests (if recurringDelay != 0) contain "hb=1".

I'm going to defer adding event handlers as there are issues with propagating some events like keyclicks.

Mental notes to self:

  • if you only send a beat when there's activity within the last n seconds, then on average, you'll expect the total time to be overstated by n/2 secs
  • to offset/compensate, you can instead send a beat if there's been activity in the last n/2 seconds

  • if there's no activity for n secs, do you stop checking for activity or stop sending beats?

  • what if activity resumes again? how long is too long ago to to resume beats?
@mattab
Owner

how do you define 'activity'? Will the ping be sent if user is looking at another tab, another window, or even away from computer?

@robocoder
Collaborator

activity: mousemove, click, mouseup, mousedown, mousewheel, scroll, DOMMouseScroll, resize, keypress, keydown, keyup, focus, and blur

  • need this many because of browser differences an the possibility of interference from other eveny handlers
  • handler simply sets the lastActivity timestamp

Yes, ping would be sent. That's why there's a check for activity. If there's no activity, I propose to no longer check.

I'll commit my changes after some more testing.

@robocoder
Collaborator

(In [3838]) refs #2041 - setHeartBeatTimer() now takes the minimumVisitLength and the heartBeatPeriod (time between pings) as parameters; this is simpler than initial ping + recurring hb; requests contain "ping=1".

also updated to jslint 2011-02-03

@robocoder
Collaborator

(In [3839]) refs #2041 - rollback buggy n/2 logic

@robocoder
Collaborator

(In [3840]) refs #2041 - undo previous optimizaton to ping on first page of a new session

@robocoder
Collaborator

(In [3841]) refs #2041 - add guard

@robocoder
Collaborator

Client-side changes are done, so it would be nice to have the server-side change done in time for 1.2 but we can roll into the next milestone.

@mattab
Owner

How often are the activity tests done? every second? Just thinking of keeping JS performance overhead low with this new feature.

I wont be able to do this in time for 1.2, so it will go in 1.3. Is pinging disabled by default? thx

@robocoder
Collaborator

Yes, disabled by default; frequency is user-defined.

@mattab
Owner

Before enabling this by default I think we could try it on some heavy websites, that are using GA, other libraries (jquery eg.) and doing Javascript heavily. Then we could see if Piwik adds any unexpected performance overhead.

For example it would be nice to say: Piwik will query a few JS variables every second to assess if the visitor is active or not, but will not conflict with any other library or Javascript and should not affect performance at all.

What do you think? for example, have you tried it on a a running piwik loading the dashboard full of widgets.

@mattab
Owner

I think, disabled by default, but easy to enable, this would make it a really cool feautre. Also, should be made available via the 'advanced' javascript tag generator see #1845

@anonymous-piwik-user

Just wondering where we are on this. I'm in the process of switching several sites from GA to Piwik, and this feature is pretty important to me. Does 1.x mean it's going to be in an indefinite release between now and 2?

@mattab
Owner

Polrbear, yes currently not roadmapped for 1.7 but it might change

@anonymous-piwik-user

I'm interessed too by this feature. I have activate it on my website, but the presentation in the server side, isn't perfect (the ping is show as a new page visit).

I hope, that it can be made for 1.7.

Thanks.

@mattab
Owner

Another bug report: http://forum.piwik.org/read.php?2,83913,page=1#msg-84060

NOTE: This feature does not work currently

@mattab
Owner

Proof Reading the upcoming book about Piwik, it's really good.

it gives me an idea related to this feature:

  • we should add a new parameter at the trackPageView function level, that users could use to trigger events/pageviews that will allow to disable the default "lower bounce rate". When "pinging" the page, we don't want to lower bounce rate by default, other users might want to do this as well in some cases.

We could allow for this other addon: send a fake Pageview 15seconds after arrival: http://analytics.blogspot.com/2012/07/tracking-adjusted-bounce-rate-in-google.html - might require the ping query so it doesn't inflate Page view counts, but changes Bounce rates for all other metrics.

@mattab
Owner

Note from Anthon:

The original thinking was, fire a ping once the page has been open some minimum amount of time, and keep pinging as long as there's some mouse activity. Basically, the goal would be to report lower bounce rates (ie if the first and only page visited) and more accurate time on site for the last page visited on a site.

The Page Visibility and traceCallback changes means the ping isn't sent until a tab gains focus and has been open the minimum amount of time. Further pings require some activity while the tab is open.

We probably need a way to restart the heartbeat if the tab regains visibility. Also, the inactivity logic doesn't work with passive activities (eg streaming media), but I think could be addressed by adding an API for players. (Are you using event tracking for this?)

This obviously isn't feature complete, so if anyone has ideas on how this should work or could work better, feel free to share your thoughts.

@mattab
Owner

I'm very interested in this feature but unfortunately time is missing to implement it for now, unless someone jumps in and produces a patch we can use! Or if you can sponsor the feature please get in touch at sponsors at piwik.org

@kevinoid

Great feature, looking forward to it. (Just commenting to add myself to the CC list.)

@mattab
Owner

See also: #5208 Detect when a user is actively browsing or using the webpage, for more accurate Time on page

@johsin18

I would really love to have this feature. There are more and more single-page applications nowadays, aren't there? For them this feature is kind of mandatory.

@RazvanMihaiu

Is this feature done? It's almost 4 years old now.

Thank you

@fredrikekelund

Just wanted to +1 this, would be a really great feature for using Piwik with SPA's

@jangrewe

another +1 here ;-)

@ru108

Try Using the Page Visibility API?

Feature Chrome (Webkit) Firefox (Gecko) Internet Explorer Opera Safari (WebKit)
Basic support 13 -webkit 10 (10) -moz 10 12.10[*] 7
33 18 (18)

http://caniuse.com/#feat=pagevisibility

@ismay

+1

@IPL01

Has this feature been implemented now?

I get a lot of single page views and they all report back as 0s and 100% bounce rates! Not good.

I am not a coder so suggesting doing something myself with the code or using an API isn't really helpful. Is there any chance something can be implemented in an upcoming update?

Many thanks.

@mattab mattab modified the milestone: Short term, Mid term
@Glisse1

Yay, short-term milestone! +1 :)

@mattab mattab modified the milestone: Piwik 2.14.0, Short term
@mattab
Owner

moved into 2.14.0 milestone, as this is such a useful feature, let's get it done :+1:

@hpvd

as already referenced above (#5208) taking existing events into account should help detect if page is only still opened in background tab without a performance problem.
=> so maybe a two step approach is possible:
1. for sites with already using events and
2. the ones which do not use events,
to be select in settings?
=> otherwise we may double effort and server load (if site alreday uses lots of events)

Another thing: do we need a "timeout" option e.g. no one looks a website longer than 15min to keep statistics fine? or 5min after last event?

@hpvd

since the function
"active life on open page / no life on page"
could be not only be used for
1) "better time on page for last page" (goal of this ticket)

but could also greatly be used as function within other features like
2) "detect page is only opened in background" details in #5845
3) "do not count automatic visits from website watchers" #3556 and #5845
4) identify unknown robots
5) ...

=> what do you think to make it a general function within Piwik?

It could bundle all technologies to decide
"active life on open page / no life on page"

  • Via triggers from events
  • via additional mouse movement tracking
  • via additional scroll tracking
  • via ping
  • via visits from same visitor on other pages at same time
  • via automatic timeout

and simply made the result available for every feature which needs it?

Of course not all technologies have to be active at the same time and one could develop a rule set
if no ...is send/received.. try ...try ...if nothing do automatic timeout

=> should we open a new ticket for this general approach? just did it #7607

@masteranalyze

I think the best solution ,is to use : Advanced Content Tracking like with Universal Analytics.
For reference i think this is the best method,that can simple solve the problem,and be quickly added on next piwik release:
http://cutroni.com/blog/2014/02/12/advanced-content-tracking-with-universal-analytics/
Original article: http://cutroni.com/blog/2012/02/21/advanced-content-tracking-with-google-analytics-part-1/

here’s what we’re going to track:

How many people scroll
When a person starts to scroll
When a person reaches the end of an article (not the end of the page, but the end of the article or post area)
When a person reaches the botton of the page (the bottom of the HTML)
Which website visitors are scanning my articles and which are reading my articles

Think about the value here! We will be able to get an accurate measure of which articles are actually read. We can even see which articles are so engaging that visitors continue through the comments to the botto of the page. Very useful stuff.

Tracking Technique

All of the above can be tracked with Event Tracking. The concept is that we will fire events when certain actions happen. Specifically we’re going to fire events based on visitors scrolling down the page.

Finally :+1:
Increase in the Average Session Duration. Events also increase the Time on Page measurement on exit pages. This may cause an increase in the Average Session Duration metric.

Increasing Bounce Rate is not Necessarily a Bad Thing!

If a visitor comes to you site, reads a post in-depth, and then leaves, should they really be classified as a bounce? Probably not. With this implementation, your bounce-rate will provide a more accurate measure of engagement and it will solve the real time spend on the website!

I hardly wait for this implemenation to be alive!

@masteranalyze

Except this events,in order to even have an more accurate time it can be added:

1)Track user that print pages :
Here is the code for google analytics that can be easily converted to piwik to track this events:
https://www.savio.no/analytics/how-to-track-printed-pages-in-google-analytics

2)Track Downloads & Other External Links,track mail to,telephone clicks.
Here is the code for google analytics that can be easily converted to piwik to track this events:
http://www.blastam.com/blog/index.php/2013/09/howto-track-downloads-links-universalanalytics

3)Social actions on page engagement:
Here is the code for google analytics that can be easily converted to piwik to track this events:
http://www.optimizesmart.com/social-interactions-tracking-through-google-analytics/

It`s working for majors:Twitter, Facebook, Google Plus & Linkedin,others can be added as need it.Maybe pininterest,stumble,etc.

If we track the 5 events from the cutroni example+3 that adds up here like :print,download,external links,and social interaction,we will be able to get an more better ideea of the interaction on that page,and not only the time will be more accurate,it will be the bounce rate too more accurate,as that user has done action and interacted with that webpage,so it should not be consider it an bounce as the user interacted with that webpage by sharing,mail to,phone,read,etc.

I think this can be implemeneted right the way with the help of the events from piwik.

@mattab
Owner

Not only have many people commented on this issue but in the forums this ticket was linked to about 40 times in the last few years.

@Glisse1

Just wanted to share the forum post from a year ago with suggestion as of what direction the visual displaying of real-time status (active, idle, reading, writing, etc) could be going... knowing the true active time on page in statistics is important, but I believe it`s just as important is to have some sort of visual representation in real-time widget

http://forum.piwik.org/read.php?3,112235,page=1#msg-112235

@masteranalyze

"Also, once this feature is implemented, a new checkbox should be added to the Advanced section in the 'Tracking Code' admin page. Checking it will generate JS code that enables this feature. "

Exactly ,this is how it should be for ping in,and for the events method,so users can simple enable from events,if they wish to enable : bounce handle,social network sharing,print,download,etc.
and once enabled,the code to automatically track them,to not have like now,to add alot of scripts for those events.
For example this:

script
setTimeout("_paq.push(['trackEvent', 'time spent', '>15sec', 'good reading'])",15000);
/script
This will make more simple to enable events and users on page time will improve and we will know it real,and also the bounce rate will be adjusted.
By solving this we will solve not only the 1st page timer ,but also the real bouncing rate,as from my point of view if an user spend 15-30 seconds or even more on just that webpage and he is interacting:sharing,printing,downloading or any action that can be done into an webpage,that is not an bounce at all.
That means the user find very usefully that blog post or whatever,and we cannot now that,unless this feature will be ready on 2.14.
Hopefully it will be alive!

@diosmosis diosmosis self-assigned this
@mattab
Owner

Summary of discussion so far:

  • piwik.js: sendRequest mechanism can be reused
  • need to decide on "ping" strategy eg. ping first time after 20s, then ping less and less often eg. [20, 40, 60, 100, 160, 260, ....] ?
  • we need to ping only when the page is actively viewed (we need to detect active tab/window somehow, maybe Page Visibility API can do this?)
  • define & document a new Tracking API parameter, eg ping=1 (or better name!)

Maybe you have more thoughts, cc @diosmosis

@hpvd

thanks for the summary. Hopefully it's ok when I would like to ask to clarify the target even more:

1) will this feature be an additional thing to other already available "site is active" signals like events (see user posts of robocoder, masteranalyze and me) so it mainly brings new active window detection?
And Piwik host will combine this and notice which signal came "last"?

2) or will it use these available signals to determine site is active, add active window detection and send a ping if there is any activity?

3) or nothing of these ;-)

Many thanks!

@mattab
Owner

@hpvd currently there is no feature - the code related to heartbeat timer / configHeartBeatTimer in piwik.js is actually unused we could/should remove it as well!

@masteranalyze

Hey guys,
Matt,hpvd,i think the best method is 1 " will this feature be an additional thing to other already available "site is active" signals like events (see user posts of robocoder, masteranalyze and me) so it mainly brings new active window detection?
And Piwik host will combine this and notice which signal came "last"? "

This is how it should be .
That way based on events,and user actions,we will have accurate TIMING+Bouncing rates!

About the : need to decide on "ping" strategy eg. ping first time after 20s, then ping less and less often eg. [20, 40, 60, 100, 160, 260, ....] ? ,ping strategy,i think from 15seconds ,to 15 seconds is okay or even from 20 seconds to 20 seconds,but this also can depends on user shared enverimonths,if the user haves Shared hosting,and this ping in eats lots of resources from the server,cpu,ram,i think their should be an recomandation for shared ,vps and dedicated server.
An dedicated server can have no problem to handle from 15 seconds to 15 seconds,but an shared server might have an problem,and maybe is good to put it from 30 or 60 seconds .
Anyhow an minimum value should be 15,and not less,and we let the user to decide.

we need to ping only when the page is actively viewed (we need to detect active tab/window somehow, maybe Page Visibility API can do this?)

Exactly,we need to ping in,only when the page is actively viewed,and this can be done with the events,that we talked more earlier.If they are no events,basically the page is maybe left in the tab for hours,and user is watching an youtube clip,so it`s not in your page tab,so basically this will just inflate the numbers witch is not good.

The events should also be there,and easily accesibile,as options in piwik,and we let the user freedom to decide,maybe some of us only needs some of the events,or all,it depends from case to case.

@matt : define & document a new Tracking API parameter, eg ping=1 (or better name!)

I think tracking api parameter, ping =1 ,is an good name,but what 1 will represent ?
1 will represent that PING is actively ?Or it will ping from 1 second to another ?
Basically 1 should Active Ping In,and 0 - Disabled Ping IN.

Hopefully we will see this in the new piwik 2.14.

@hpvd

"1) = additional thing to other already available "site is active" signals"
that's also my preferred way, and was the reason for creating #7607

(maybe one should taking into account the date this ticket about supporting ping requests was created/ started: "21 Jan 2011" - at this time there was no great "event tracking" available within piwik)

@masteranalyze

so this will work,basically on 15 seconds to 15 seconds to the end user pings?
or via events triggers ?

I agree,hpvd,there is no great "Event tracking" available in piwik.

@hpvd

@masteranalyze now standard event tracking is fine for me.
What I was describing was the time the ticket was started...in 2011 there was no event tracking at all: so you had no chance to determine if there is someone on the site.
If you are using event tracking today, you got lots of activity signals from each site during the usage by the visitor. So the ping only needs to be an additional possibility to determine if there is activity.
But of course the signals from the events first have to be included in the determination of active/passive...
@all <= do we need an extra ticket for this??
or should it stay be part of #7607?

@masteranalyze

@hpvd is clearly that from that time to today,piwik did have an great evolution,but i think it can be even more great,as except the fact the anaytics are private,there is one more good thing and an big advantage for PIWIK - Server Side Tracking,witch is not possible neither in Google Analytics,neither in StatCounter,neither in Clicky or any other stats available on the market today.

The event tracking we have today,of course is more good,that what it was missing in 2011,i totally agree with you,but nowdays for every event tracking,you need to add one more line code into the javascript,one more for another event and so on,it will be alot more easily,if in the event tracking back -end it will be events with options : YES ,NO ,and addiacently the options to it,so everybody can Enable or use them,right now Events are not quite so easily setup for someone who does not write code.
This is for example for : ADJUSTED BOUNCED RATE,but if you dont add it to your code of tracking,it wont track this event,and mostly your bounce rate will be higher,and time spend on the page less.
script
setTimeout("_paq.push(['trackEvent', 'time spent', '>15sec', 'good reading'])",15000);
/script
Maybe we can make an list of all this,scripts for events how they should be,and further on,they can be implemented and be available from backend Event Tracking!

"If you are using event tracking today, you got lots of activity signals from each site during the usage by the visitor." - i totally agree with you,with the pointing of that only people who setup the events and puts additionally script,can track the events,or if the software your using,haves integer piwik,you will have those events,else not.

" So the ping only needs to be an additional possibility to determine if there is activity.
But of course the signals from the events first have to be included in the determination of active/passive..." - like i said and i agree with you,that this ping in,normaly should be an option from backend,that everybody to be able to setup up,for example on 15 seconds,on 30seconds,to be the ping in .
And we also do needs all the Events,in Back END,to check the user activity,if he is active or passive.
In administration of piwik,we don`t have Events ,to be easily setup,with yes or no,and to not have to add,more and more code,into the piwik original tracking code.

hpvd,create an new ticket for PIWIK EVENTS SCRIPTS,and there we can put an list of EVENTS,and the accordingly script,but afterwards,how we manage to put them into the backend of piwik?!
What will be the tracking script for this action : Social actions on page engagement:
Here is the code for google analytics that can be easily converted to piwik to track this events:
http://www.optimizesmart.com/social-interactions-tracking-through-google-analytics/ ?!
I think if we all contribute and make an list,afterwards,that list can be complected along the time!

@hpvd

@masteranalyze
a ticket for collecting standard events and the js code snippets belonging to them
is a great idea for a future improvement - please start the ticket!!

@masteranalyze

@hpvd i opened this #8141 ,for posting colection of standard events and js code.
The thing is,how we,do implement them in piwik back-end to be easily accesibile?Even for non-coders.

@diosmosis
Collaborator

Fixed in #8069.

@diosmosis diosmosis closed this
@robocoder
Collaborator
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.