New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use localStorage for Google Analytics tracking when available #1444

Closed
davidmurdoch opened this Issue Oct 7, 2013 · 30 comments

Comments

Projects
None yet
8 participants
@davidmurdoch
Contributor

davidmurdoch commented Oct 7, 2013

TL;DR:

(function(b,o,i,l,e,r){b.GoogleAnalyticsObject=l;b[l]||(b[l]=
function(){(b[l].q=b[l].q||[]).push(arguments)});b[l].l=+new Date;
e=o.createElement(i);r=o.getElementsByTagName(i)[0];
e.src='//www.google-analytics.com/analytics.js';
r.parentNode.insertBefore(e,r)}(window,document,'script','ga'));
ga('create','UA-XXXXX-X',{'storage': 'none','clientId':localStorage.getItem('gaClientId')});
ga(function(t){localStorage.setItem('gaClientId',t.get('clientId'));});
ga('send','pageview');

The Source:
http://stackoverflow.com/questions/4502128/convert-google-analytics-cookies-to-local-session-storage/19207035?noredirect=1#19207035

Google Analytics Docs:
https://developers.google.com/analytics/devguides/collection/analyticsjs/domains#disableCookies

We could use Modernizer.localstorage to check for localStorage support and fallback to cookies if its not available. Though I'm not sure if we want to lock in Modernizr as a dependency.

Why?
Because Google doesn't need to send their cookie to your server for every single request to your domain (or theirs, for that matter).

@alrra

This comment has been minimized.

Member

alrra commented Oct 8, 2013

Though I'm not sure if we want to lock in Modernizr as a dependency.

Maybe it would be best to just add it to the docs ?

Also, ping @mathiasbynens.

@drublic

This comment has been minimized.

Member

drublic commented Oct 8, 2013

Thanks for optimizing the snipped, David. As @alrra I think we are good with adding it to the docs.

@davidmurdoch

This comment has been minimized.

Contributor

davidmurdoch commented Oct 8, 2013

The credit doesn't belong to me; this was brought to my attention by @elmerbulthuis. Though I wouldn't really consider this an optimization of the snippet itself, per se --- it is more of an optimization of the web as a whole :-p.

I wonder how many bytes could be saved, globally, if everyone adopted the localStorage solution.

@mathiasbynens

This comment has been minimized.

Member

mathiasbynens commented Oct 8, 2013

I’m obviously a big fan of this solution. The only problem with including it by default in the boilerplate is the one @davidmurdoch mentioned: we need to feature-test for localStorage first. This can be done by using Modernizr or by adding a small piece of standalone code but either way it’s going to slightly increase the page size. Then again, it will save many bytes in the long run, since no cookies will be sent in the request headers for any resources on the affected domain.

@mathiasbynens

This comment has been minimized.

Member

mathiasbynens commented Oct 8, 2013

Something like this:

(function(b,o,i,l,e,r){b.GoogleAnalyticsObject=l;b[l]||(b[l]=
function(){(b[l].q=b[l].q||[]).push(arguments)});b[l].l=+new Date;
e=o.createElement(i);r=o.getElementsByTagName(i)[0];
e.src='//www.google-analytics.com/analytics.js';
r.parentNode.insertBefore(e,r)}(window,document,'script','ga'));
(function(){var a=(function(){var c=new Date,b;try{
localStorage.setItem(c,c);b=localStorage.getItem(c)==c;
localStorage.removeItem(c);return b&&localStorage}catch(d){}}());
ga('create','UA-XXXXX-X',a?{storage:'none',clientId:a.gaId}:{});
ga(function(b){a.gaId=b.get('clientId')});ga('send','pageview')}());

(It uses the localStorage feature test taken from http://mathiasbynens.be/notes/localstorage-pattern.)

It seems to work fine after some quick tests. I’ll create a PR after testing this more extensively. (Help welcome, of course!)

@davidmurdoch

This comment has been minimized.

Contributor

davidmurdoch commented Oct 8, 2013

FYI: this has the potential to save about 33 raw bytes (headers/cookies aren't compressed) per round-trip for each request to the affected domains.

@mathiasbynens' current inline-feature-detect solution is 130 compressed bytes∗ larger (obviously this will be different for each unique page, but it gives us a rough idea). So, we should probably see if we can golf this down a bit more.

I'd personally like to see the gzipped diff down to a 65 bytes and will give it a shot myself soon. :-)

∗using this deflator: http://www.vervestudios.co/projects/compression-tests/snippet-deflator

@davidmurdoch

This comment has been minimized.

Contributor

davidmurdoch commented Oct 14, 2013

318 GZIPped bytes (our current version is 248 GZIPped bytes):

(function(l,e){GoogleAnalyticsObject='ga',(window.ga||(ga=function(l,e){(ga.q=ga.q||[]).push(arguments)})).l=+new Date,l=document.createElement('script'),l.src='//www.google-analytics.com/analytics.js',(e=document.getElementsByTagName('script')[0]).parentNode.insertBefore(l,e);ga('create','UA-XXXXX-X',(function(l,e){try{l=(localStorage[ga.l]=ga.l)==ga.l;localStorage.removeItem(ga.l);return l}catch(l){}}())?{storage:'none',clientId:localStorage.clientId}:{});ga(function(l,e){localStorage.clientId=l.get('clientId')});ga('send','pageview')}())

This isn't tested very well, so I'll still need to do that. But its a start.

And unfortunately, the localStorage test is probably compromised in some browser somewhere, since I got rid of setItem and getItem calls and used some other golfing "tricks".

That's all I got for now. :-)

@mathiasbynens

This comment has been minimized.

Member

mathiasbynens commented Oct 15, 2013

It just occurred to me we’ve been gzipping the snippet itself, which is kinda pointless. Gzip results depend on the rest of the document (i.e. the HTML source if it’s inlined in a document, or the rest of the JavaScript file if it’s part of one). Maybe comparing gzipped sizes of just the snippet is not the best way to measure this?

Your snippet looks nice. Good catch re-using the ga.l timestamp instead of generating a new one!

And unfortunately, the localStorage test is probably compromised in some browser somewhere, since I got rid of setItem and getItem calls and used some other golfing "tricks".

If that’s the case, it would be a dealbreaker IMHO.

We can replace document.getElementsByTagName('script')[0] with document.scripts[0] when Firefox < 9 support is not an issue anymore.

@davidmurdoch

This comment has been minimized.

Contributor

davidmurdoch commented Oct 15, 2013

@mathiasbynens GZIPping just the snippet will approximate the minimum byte savings from compression. So it is not entirely a moot point. In nearly all cases, the compression ratio for the snippet will increase as the page size increases.

Still need to test! I've added the getItem and setItem calls back in and managed to get it down to 309 bytes:

+function(l,e){(ct=this[GoogleAnalyticsObject='ct']||function(l,e){(ct.q=ct.q||[]).push(arguments)}).l=+new Date,l=document.createElement('script'),l.src='//www.google-analytics.com/analytics.js',(e=document.getElementsByTagName('script')[0]).parentNode.insertBefore(l,e);try{localStorage.setItem(ct.l,ct.l),l=localStorage.getItem(ct.l)-ct.l,localStorage.removeItem(ct.l)}catch(l){};ct('create','UA-XXXXX-X',l?{}:{clientId:localStorage.clientId,storage:'none'}),ct(function(l,e){localStorage.clientId=l.get('clientId')}),ct('send','pageview')}()
  • I'm now using an IIFE that uses a + sign instead of wrapping parentheses.
  • I'm also using localStorage.clientId instead of localStorage.gaId as clientId saves some bytes.
  • Using this instead of window saved 1 more byte (in combination with moving the GoogleAnalyticsObject assignment).
  • Changing ga to ct (ct is more prevalent) saved one more byte (this is probably not worth the confusion).
  • Getting rid of the function call and reusing l for the localStorage check by assigning it to 0 on success saved a bunch of bytes.

Again, this needs lots more testing.

@drublic

This comment has been minimized.

Member

drublic commented Dec 15, 2013

@davidmurdoch Any updates on the tests yet? Can we write down a test flow for this so others can help test?

@davidmurdoch

This comment has been minimized.

Contributor

davidmurdoch commented Jan 17, 2014

Sorry i've been MIA, I got put on a high-priority, 6 month project and haven't been able to devote much time to anything else.

The easiest (and dumbest) way to test this is to just replace your analytics code with this new code and see if you get any odd fluctuations in numbers and browsers versions. I've done this myself and haven't seen anything that sticks out. However, I don't' have many oldie visitors anyway.

Another way would be to load this experimental analytics script in a generated iframe (so as not to interfere with the stable analytics snippet) and call _trackPageview from there, under a different GA account, of course. Then you just need to compare the data after a week or so.

I can't promise that I can work on a drop-in snippet for testing this any time soon; if someone else wants to take ownership of these ideas while I go back into hiding please go right ahead. :-)

drublic added a commit to drublic/css-modal that referenced this issue Mar 12, 2014

@drublic

This comment has been minimized.

Member

drublic commented Mar 12, 2014

I've just started a test for http://drublic.github.io/css-modal/. I got 97k page views last months but wildly spread around browser.

The numbers:

  1. Chrome 44.01%
  2. Firefox 34.38%
  3. Internet Explorer 8.86%
  4. Opera 5.26%
  5. Safari 4.01%
  6. Android Browser 2.22%

Let's wait and see. I got the "normal" statistics running in parallel.

Apart from that I think the code needs some more updates for readability (80 chars per line and where to insert the identifier).

I'll get back to this test in around a week.

@drublic

This comment has been minimized.

Member

drublic commented Mar 17, 2014

I am a bit early but my findings are pretty stable at the moment. Unfortunately I see big difference in the number of visitors for both accounts.

The default implementation shows 2,964 unique visits for March 13th to 17th.
The local storage based shows 756 unique visits for the same time frame.

There might be three possible reasons:

  • my implementation of the snipped is corrupted
  • loading the iframe is blocked by browsers
  • the local storage integration of the snipped is broken

Currently I don't see any mistakes in my code here: http://drublic.github.io/css-modal/test-gau-localstorage.html (which is the iframe that has been integrated in the site).

Also I haven't experienced iframes being blocked by browsers or pages. Has anyone an idea if this might happen?

Which leads me to the solution that the local storage GUA snipped has bugs. I have not looked into what the problems might be.
Can we develop an unminifed variant for further testing and minimize after we could a working solution?

Also I'd opt for descoping this from HTML5BP v5.0 and release it with 5.1 if we find a solution. What do you guys think?

@alrra alrra added this to the 5.1.0 milestone Mar 17, 2014

@alrra

This comment has been minimized.

Member

alrra commented Mar 17, 2014

Also I'd opt for descoping this from HTML5BP v5.0 and release it with 5.1

@drublic 👍 (added issue to the v5.1.0 milestone).

@tomfuertes

This comment has been minimized.

Contributor

tomfuertes commented Jun 8, 2014

If you're numbers are that off it's probably the fact you have to supply a default clientId when calling ga('create', w/ storage:'none'.

https://developers.google.com/analytics/devguides/collection/analyticsjs/domains#disableCookies

@davidmurdoch

This comment has been minimized.

Contributor

davidmurdoch commented Sep 22, 2014

Just blogged about this issue on my site, here: Google Async Analytics using LocalStorage and set up a test page here: http://davidmurdoch.com/google-async-analytics-using-localstorage-test/.

Please read, share, and test.

(note: if you find any typos or errors on those pages, let me know over on twitter @pxcoach.

@philipwalton

This comment has been minimized.

Contributor

philipwalton commented Sep 29, 2014

Hey, sorry to get here a little bit late to the party. I work on the Google Analytics team, and I wanted to comment and offer my thoughts on this issue.

First of all, I don't think it's a good idea for the H5BP project to recommend a Google Analytics tracking snippet that's functionally different from the officially recommended one. People will probably assume they're the same, and if they're actually not, it'll cause confusion. If the Google Analytics documentation claims GA supports some feature and it doesn't because someone's using a different snippet, that will probably lead to some pretty hard to debug issues (especially if H5BP doesn't make it obvious that the snippets are different).

If there's something that GA could do better, we'd love to evolve with the needs of the community rather than diverge from it. (BTW, feel free to ping or cc me on any GA related Github issues.)

Anyway, here's the main problem with localStorage and why GA doesn't offer it as the default storage mechanism:

localStorage is scoped to location.origin whereas cookies can be scoped to a top level domain. Cookie storage allows analytics.js to do subdomain tracking out of the box, and this wouldn't be possible with localStorage. In addition, if parts of your site are HTTP and other parts are HTTPS, that would also fail (and by fail I mean the storage isn't shared, so you'd lose the client ID and GA would treat it as a separate session). While it's true that these aren't concerns for most GA users, I still think it'd be bad to offer this proposed snippet as a drop-in-replacement due to the feature-loss I just described.

That being said, based on this issue and @davidmurdoch's blog post, we're going to try to prioritize building an officially supported localStorage mechanism. Currently the storage parameter only supports the options cookie and none, but we'd like to add a third localStorage option, so users who don't need subdomain or cross-scheme tracking can opt in. I don't know when this will be added, but I can update this issue when/if it is.

Does this seem reasonable to everyone?

@alrra

This comment has been minimized.

Member

alrra commented Sep 30, 2014

@philipwalton Thanks for the comment!

Does this seem reasonable to everyone?

Cc: @davidmurdoch, @mathiasbynens

@mathiasbynens

This comment has been minimized.

Member

mathiasbynens commented Sep 30, 2014

That being said, based on this issue and @davidmurdoch's blog post, we're going to try to prioritize building an officially supported localStorage mechanism.

👍

Please update the issue when this gets added. Thanks!

@davidmurdoch

This comment has been minimized.

Contributor

davidmurdoch commented Sep 30, 2014

@philipwalton, 👍 Excellent news! However, you don't need to try to build it, we already did! :-p (I kid, I kid).

I'll go ahead and update my blog post with this news, and create a GitHub repo with the unofficial localStorage tracking code, making sure to emphasize its shortcomings. Thanks!

@jonathantneal

This comment has been minimized.

Member

jonathantneal commented Sep 30, 2014

👍 but it also seems like the future web needs some kind of topLevelStorage. Glad the option will be made available. With that in mind, and when the snippet gets in, what might the preference be for h5bp?

@davidmurdoch

This comment has been minimized.

Contributor

davidmurdoch commented Sep 30, 2014

@jonathantneal, we had globalStorage in Firefox, which did cross scheme, port, and sub-domain storage. Firefox was the only one to implement it, and it has since been marked obsolete. :-(

@alrra alrra referenced this issue Oct 10, 2014

Closed

Add the code, a short description, and links! #1

3 of 3 tasks complete
@alrra

This comment has been minimized.

Member

alrra commented Oct 10, 2014

@davidmurdoch Thank you so much for opening this issue and digging into it, we sincerely appreciate it!

@philipwalton Thanks again for joining the discussion, and like @mathiasbynens said, please keep us updated!

@alrra alrra closed this Oct 10, 2014

@alrra

This comment has been minimized.

Member

alrra commented Oct 10, 2014

and create a GitHub repo with the unofficial localStorage tracking code, making sure to emphasize its shortcomings.

@davidmurdoch's repository is https://github.com/davidmurdoch/ga-localstorage (although it is not yet updated).

@davidmurdoch

This comment has been minimized.

Contributor

davidmurdoch commented Oct 15, 2014

I just published the "Google Analytics using localStorage" script to npm: https://www.npmjs.org/package/ga-localstorage

The https://github.com/davidmurdoch/ga-localstorage repo has also been updated with the code.

ghost referenced this issue in roots/soil Mar 13, 2015

@caesarsol

This comment has been minimized.

caesarsol commented Jun 4, 2015

@philipwalton

This comment has been minimized.

Contributor

philipwalton commented Jun 4, 2015

@caesarsol I think that's a really bad idea. As I described in my comment, cookies and localStorage don't have the same restrictions, so swapping them out for every single script that runs on the page is extremely risky.

@caesarsol

This comment has been minimized.

caesarsol commented Jun 4, 2015

hello @philipwalton, thanks for the response but maybe I explained poorly, i was referring to this comment by SO user smhmic:

This might violate GA TOS! Here is a secondhand quote from a GA Team member, taken from this article: "Using HTTP State Management mechanisms" (read: localStorage) "to propagate cookie state is a circumvention of our privacy safeguards. Doing so violates the Google Analytics Terms of Service". My interpretation of this is that GA employs cookies and not localStorage because more users are familiar with the concept of cookies and how to clear them; thus, GA's use of cookies is a privacy feature. – smhmic

@philipwalton

This comment has been minimized.

Contributor

philipwalton commented Jun 4, 2015

Using HTTP State Management mechanisms" (read: localStorage) to propagate cookie state is a circumvention of our privacy safeguards. Doing so violates the Google Analytics Terms of Service

Hmmm, I don't think this is true. There are opt-out features that GA provides (e.g. Chrome extensions) that don't rely on the implementor using cookies. I think the point of this section of the TOS is that you can't create a mechanism by which someone who's using an official "do not track" extension will still be tracked.

I can look into it further and I'll update this thread is my assumptions turn out to be false.

@davidmurdoch

This comment has been minimized.

Contributor

davidmurdoch commented Apr 21, 2016

Update:

It is not against TOS to use localStorage to store the ClientID; it is now officially supported by Google: https://developers.google.com/analytics/devguides/collection/analyticsjs/cookies-user-id#using_localstorage_to_store_the_client_id

Note: if you have to support (extremely) old browsers (like iOS5 and FF4) their example snippet may fail (see: https://github.com/Modernizr/Modernizr/blob/master/feature-detects/storage/localstorage.js).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment